[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:31 dist-test-slave-1t6p krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:31 dist-test-slave-1t6p krb5kdc[395](info): set up 2 sockets
Sep 27 05:12:31 dist-test-slave-1t6p krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 05:12:32.220170 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:12:32 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:32.378100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45673
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:32.682891 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:32.683207 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:32.683367 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:32.690574 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:32.690665 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:32.690765 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:32.690865 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:32.690948 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:32.700059 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:32.701236 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:32.702495 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:32.713879 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:32.713984 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:32.713959 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:32.714936 408 server_base.cc:1034] running on GCE node
I20260927 05:12:32.715695 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:32.716862 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:32.718065 408 hybrid_clock.cc:648] HybridClock initialized: now 1790485952717988 us; error 69 us; skew 500 ppm
I20260927 05:12:32.718381 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:32 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:32.721669 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:32.722968 408 webserver.cc:460] Webserver started at http://127.0.97.62:40815/ using document root <none> and password file <none>
I20260927 05:12:32.723613 408 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:32.723703 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:32.723923 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:32.726590 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "d295fdffa50f4809b255607131db8006"
format_stamp: "Formatted at 2026-09-27 05:12:32 on dist-test-slave-1t6p"
server_key: "f667534f12651b892f07733add8dbf17"
server_key_iv: "2488fe0bf2ca35036ebaae396e19dad6"
server_key_version: "encryptionkey@0"
I20260927 05:12:32.727319 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "d295fdffa50f4809b255607131db8006"
format_stamp: "Formatted at 2026-09-27 05:12:32 on dist-test-slave-1t6p"
server_key: "f667534f12651b892f07733add8dbf17"
server_key_iv: "2488fe0bf2ca35036ebaae396e19dad6"
server_key_version: "encryptionkey@0"
I20260927 05:12:32.732084 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:12:32.735196 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:32.736394 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:32.736577 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "d295fdffa50f4809b255607131db8006"
format_stamp: "Formatted at 2026-09-27 05:12:32 on dist-test-slave-1t6p"
server_key: "f667534f12651b892f07733add8dbf17"
server_key_iv: "2488fe0bf2ca35036ebaae396e19dad6"
server_key_version: "encryptionkey@0"
I20260927 05:12:32.736728 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:32.770951 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:32.774638 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:32.774916 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:32.794823 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45673
I20260927 05:12:32.794843 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45673 every 8 connection(s)
I20260927 05:12:32.796257 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:12:32.799995 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:32.802405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 408
I20260927 05:12:32.802708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:12:32.805097 388 external_mini_cluster.cc:1428] Setting key dc4d7965384f31a3052d5910f7a7953d
I20260927 05:12:32.812860 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006: Bootstrap starting.
I20260927 05:12:32.815670 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:32.816582 475 log.cc:824] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:32.819103 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006: No bootstrap required, opened a new log
I20260927 05:12:32.826761 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } }
I20260927 05:12:32.827133 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:32.827220 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d295fdffa50f4809b255607131db8006, State: Initialized, Role: FOLLOWER
I20260927 05:12:32.827821 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } }
I20260927 05:12:32.828058 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:12:32.828234 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:12:32.828405 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:32.831086 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } }
I20260927 05:12:32.831557 475 leader_election.cc:302] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d295fdffa50f4809b255607131db8006; no voters:
I20260927 05:12:32.832154 475 leader_election.cc:288] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:12:32.832288 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 05:12:32 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:32.832870 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [term 1 LEADER]: Becoming Leader. State: Replica: d295fdffa50f4809b255607131db8006, State: Running, Role: LEADER
I20260927 05:12:32.833406 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } }
I20260927 05:12:32.834604 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:12:32.836117 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d295fdffa50f4809b255607131db8006. Latest consensus state: current_term: 1 leader_uuid: "d295fdffa50f4809b255607131db8006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } } }
I20260927 05:12:32.836560 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:32.841390 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d295fdffa50f4809b255607131db8006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d295fdffa50f4809b255607131db8006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45673 } } }
I20260927 05:12:32.841728 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:32.842299 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:12:32.847834 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:32.809450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.0.97.62:45673)
0927 05:12:32.810473 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:32.810489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:32.812251 (+ 1762us) server_negotiation.cc:407] Connection header received
0927 05:12:32.821039 (+ 8788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:32.821056 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:32.821398 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:32.821767 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:32.824470 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:32.825787 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:32.827529 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:32.828328 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:32.828511 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:32.834441 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:32.834448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:32.834460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:32.834491 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:32.839436 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:32.840731 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:32.840738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:32.840744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:32.840938 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:32.841527 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:32.841534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:32.841542 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:32.842317 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:32.845354 (+ 3037us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:32.845686 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 05:12:32.846199 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":504,"threads_started":1}
I20260927 05:12:32.853345 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:12:32.863317 485 catalog_manager.cc:1347] Generated new cluster ID: 2030aeb1691646a2bbbc49b4328f718e
I20260927 05:12:32.863438 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:12:32.876057 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:12:32.878230 485 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:12:32.891783 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d295fdffa50f4809b255607131db8006: Generated new TSK 0
I20260927 05:12:32.892812 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:12:32.937852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:33.261942 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:33.262217 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:33.262382 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:33.269497 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:33.269588 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:33.269775 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:12:33.278970 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:33.280561 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:33.281807 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:33.294142 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:33.294080 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:33.294170 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:33.294991 501 server_base.cc:1034] running on GCE node
I20260927 05:12:33.295598 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:33.296404 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:33.297628 501 hybrid_clock.cc:648] HybridClock initialized: now 1790485953297587 us; error 60 us; skew 500 ppm
I20260927 05:12:33.297925 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:33 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:33.301231 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:12:33.302603 501 webserver.cc:460] Webserver started at http://127.0.97.1:43851/ using document root <none> and password file <none>
I20260927 05:12:33.303320 501 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:33.303453 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:33.303743 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:33.306473 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "e04eb5ac6d0c4482ae21f99f79aad3eb"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "e43c8c05dfc5aeff7d1c4228c04dc3e7"
server_key_iv: "9204e618e93b5ce4f8de16c8103e0a53"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.307301 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e04eb5ac6d0c4482ae21f99f79aad3eb"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "e43c8c05dfc5aeff7d1c4228c04dc3e7"
server_key_iv: "9204e618e93b5ce4f8de16c8103e0a53"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.312175 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:12:33.315250 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.316462 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:12:33.316658 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "e04eb5ac6d0c4482ae21f99f79aad3eb"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "e43c8c05dfc5aeff7d1c4228c04dc3e7"
server_key_iv: "9204e618e93b5ce4f8de16c8103e0a53"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.316847 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:33.339898 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:33.344658 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:33.344926 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:33.345837 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:33.347085 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:33.347188 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.347324 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:33.347393 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.382683 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41195
I20260927 05:12:33.382719 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41195 every 8 connection(s)
I20260927 05:12:33.384023 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:12:33.393337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 501
I20260927 05:12:33.393576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:12:33.394227 388 external_mini_cluster.cc:1428] Setting key ce16a62ff5ef84d557366802ea67e9cd
Sep 27 05:12:33 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:33.404732 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:33.386435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57331 (local address 127.0.97.62:45673)
0927 05:12:33.386860 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:33.386865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:33.387877 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 05:12:33.389691 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:33.389698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:33.389797 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:33.389914 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:33.391564 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.392314 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.394319 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.394778 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.394929 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:33.398433 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:33.398442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:33.398450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:33.398494 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:33.401601 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:33.402508 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:33.402512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:33.402514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:33.402621 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:33.403124 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:33.403128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:33.403130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:33.403364 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:33.403469 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:33.404335 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 05:12:33.404489 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260927 05:12:33.406316 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:33.386981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45673 (local address 127.0.97.1:57331)
0927 05:12:33.387720 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:33.387760 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:33.389427 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:33.390151 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:33.390159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:33.390779 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:33.391395 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:33.391412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.392467 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.392478 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:33.394172 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:33.394184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.395058 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.395067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:33.395368 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:33.396126 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:33.396147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:33.398178 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:33.401740 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:33.401751 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:33.401770 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:33.402387 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:33.402724 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:33.402732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:33.402738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:33.402988 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:33.403485 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:33.403496 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:33.403937 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 05:12:33.404251 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 05:12:33.404822 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":265,"threads_started":1}
I20260927 05:12:33.407931 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45673
I20260927 05:12:33.408226 629 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:33.408893 629 heartbeater.cc:511] Master 127.0.97.62:45673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:12:33.410920 440 ts_manager.cc:194] Registered new tserver with Master: e04eb5ac6d0c4482ae21f99f79aad3eb (127.0.97.1:41195)
I20260927 05:12:33.412436 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:12:33.433986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:33.746920 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:33.747220 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:33.747387 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:33.754439 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:33.754611 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:33.754809 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:12:33.764622 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:33.765873 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:33.767171 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:33.779942 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:33.780066 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:33.780088 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:33.781067 635 server_base.cc:1034] running on GCE node
I20260927 05:12:33.781713 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:33.782560 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:33.783754 635 hybrid_clock.cc:648] HybridClock initialized: now 1790485953783708 us; error 39 us; skew 500 ppm
I20260927 05:12:33.784042 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:33 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:33.786972 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:12:33.788383 635 webserver.cc:460] Webserver started at http://127.0.97.2:36787/ using document root <none> and password file <none>
I20260927 05:12:33.789016 635 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:33.789153 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:33.789449 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:33.792579 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "086397e2cb594028911f356db19217f8"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "14be295a45e12dd3f28906e5d962797c"
server_key_iv: "3d08f7def117b5f37fc20c5a80a3f8b9"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.793320 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "086397e2cb594028911f356db19217f8"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "14be295a45e12dd3f28906e5d962797c"
server_key_iv: "3d08f7def117b5f37fc20c5a80a3f8b9"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.798296 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 05:12:33.801604 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.802915 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:12:33.803087 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "086397e2cb594028911f356db19217f8"
format_stamp: "Formatted at 2026-09-27 05:12:33 on dist-test-slave-1t6p"
server_key: "14be295a45e12dd3f28906e5d962797c"
server_key_iv: "3d08f7def117b5f37fc20c5a80a3f8b9"
server_key_version: "encryptionkey@0"
I20260927 05:12:33.803248 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:33.828455 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:33.832738 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:33.833014 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:33.833914 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:33.835863 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:33.835937 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.836078 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:33.836138 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:33.874886 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34469
I20260927 05:12:33.874933 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34469 every 8 connection(s)
I20260927 05:12:33.876191 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:12:33.878302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 635
I20260927 05:12:33.878499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:12:33.878942 388 external_mini_cluster.cc:1428] Setting key 3e9403706fcb07f9d8a32ccff3485356
Sep 27 05:12:33 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:12:33.897884 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:33.879344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57709 (local address 127.0.97.62:45673)
0927 05:12:33.879531 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:33.879537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:33.881034 (+ 1497us) server_negotiation.cc:407] Connection header received
0927 05:12:33.882946 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:33.882954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:33.883044 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:33.883178 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:33.884646 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.885363 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.886924 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.887397 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.887530 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:33.891674 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:33.891680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:33.891685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:33.891723 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:33.894489 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:33.895273 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:33.895279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:33.895282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:33.895407 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:33.895807 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:33.895814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:33.895817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:33.896063 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:33.896214 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:33.896895 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 05:12:33.897115 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:12:33.898910 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:33.879939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45673 (local address 127.0.97.2:57709)
0927 05:12:33.880871 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:33.880921 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:33.882608 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:33.883357 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:33.883364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:33.883913 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:33.884468 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:33.884484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.885535 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.885539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:33.886769 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:33.886779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:33.887624 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:33.887638 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:33.887849 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:33.888522 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:33.888544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:33.891442 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:33.894633 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:33.894642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:33.894659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:33.895156 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:33.895511 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:33.895517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:33.895521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:33.895700 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:33.896191 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:33.896199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:33.896542 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 05:12:33.896826 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 05:12:33.897341 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":426,"threads_started":1}
I20260927 05:12:33.900266 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45673
I20260927 05:12:33.900557 763 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:33.901253 763 heartbeater.cc:511] Master 127.0.97.62:45673 requested a full tablet report, sending...
I20260927 05:12:33.902583 440 ts_manager.cc:194] Registered new tserver with Master: 086397e2cb594028911f356db19217f8 (127.0.97.2:34469)
I20260927 05:12:33.903470 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:12:34.416090 629 heartbeater.cc:503] Master 127.0.97.62:45673 was elected leader, sending a full tablet report...
I20260927 05:12:34.906596 763 heartbeater.cc:503] Master 127.0.97.62:45673 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260927 05:12:34.975785 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 1.094s user 0.001s sys 0.007s
I20260927 05:12:34.976509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:35.313015 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:35.313315 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:35.313473 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:35.320465 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:35.320576 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:35.320755 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:12:35.330397 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:35.331607 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:35.332849 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:35.346561 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:35.346593 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:35.346625 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:35.347281 768 server_base.cc:1034] running on GCE node
I20260927 05:12:35.347911 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:35.348760 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:35.349985 768 hybrid_clock.cc:648] HybridClock initialized: now 1790485955349951 us; error 30 us; skew 500 ppm
I20260927 05:12:35.350320 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:35.354192 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:12:35.355567 768 webserver.cc:460] Webserver started at http://127.0.97.3:32995/ using document root <none> and password file <none>
I20260927 05:12:35.356179 768 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:35.356276 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:35.356511 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:35.359242 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "f456f04a5ed9475ebddba174e1d1bb6a"
format_stamp: "Formatted at 2026-09-27 05:12:35 on dist-test-slave-1t6p"
server_key: "1c8ba9b43404ffea1da465bad560e6f6"
server_key_iv: "94649c617f71c8ae95400337baa10131"
server_key_version: "encryptionkey@0"
I20260927 05:12:35.359949 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f456f04a5ed9475ebddba174e1d1bb6a"
format_stamp: "Formatted at 2026-09-27 05:12:35 on dist-test-slave-1t6p"
server_key: "1c8ba9b43404ffea1da465bad560e6f6"
server_key_iv: "94649c617f71c8ae95400337baa10131"
server_key_version: "encryptionkey@0"
I20260927 05:12:35.365149 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:12:35.368403 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:35.369649 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:12:35.369774 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "f456f04a5ed9475ebddba174e1d1bb6a"
format_stamp: "Formatted at 2026-09-27 05:12:35 on dist-test-slave-1t6p"
server_key: "1c8ba9b43404ffea1da465bad560e6f6"
server_key_iv: "94649c617f71c8ae95400337baa10131"
server_key_version: "encryptionkey@0"
I20260927 05:12:35.369921 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:35.382381 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:35.386104 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:35.386364 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:35.387286 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:35.388445 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:35.388513 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:35.388679 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:35.388727 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:35.428906 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34279
I20260927 05:12:35.428937 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34279 every 8 connection(s)
I20260927 05:12:35.430217 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:12:35.432396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 768
I20260927 05:12:35.432559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:12:35.433156 388 external_mini_cluster.cc:1428] Setting key 36a1839e1e2ed5c0378e4f90ff4accdc
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:35.450286 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.432814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38519 (local address 127.0.97.62:45673)
0927 05:12:35.433729 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.433736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.434801 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 05:12:35.437057 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.437060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.437116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.437204 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.438654 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.439515 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.441026 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.441413 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.441512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.444843 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.444847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.444850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.444884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.447368 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.448196 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.448203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.448206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.448348 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.448816 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.448822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.448825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.449110 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.449219 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.449876 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.450025 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":672,"threads_started":1}
I20260927 05:12:35.451157 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.433790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45673 (local address 127.0.97.3:38519)
0927 05:12:35.434622 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:35.434672 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:35.436756 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:35.437440 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:35.437448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:35.437986 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:35.438515 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:35.438531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.439654 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.439658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:35.440881 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:35.440892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.441591 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.441595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:35.441823 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.442491 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:35.442512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:35.444653 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:35.447503 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:35.447511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:35.447529 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:35.448078 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:35.448456 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:35.448460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:35.448463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:35.448654 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:35.449223 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:35.449233 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:35.449514 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 05:12:35.449732 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 05:12:35.450069 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":322,"threads_started":1}
I20260927 05:12:35.452921 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45673
I20260927 05:12:35.453186 896 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:35.453804 896 heartbeater.cc:511] Master 127.0.97.62:45673 requested a full tablet report, sending...
I20260927 05:12:35.455259 440 ts_manager.cc:194] Registered new tserver with Master: f456f04a5ed9475ebddba174e1d1bb6a (127.0.97.3:34279)
I20260927 05:12:35.456043 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38519
I20260927 05:12:35.460876 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:35.537483 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.521568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.0.97.1:41195)
0927 05:12:35.522826 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.522833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.522860 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:12:35.522919 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.522926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.523199 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.523353 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.524665 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.525688 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.526893 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.527510 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.527607 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.531857 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.531867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.531881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.531917 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.534973 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.535688 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.535696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.535698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.535815 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.536210 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.536213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.536215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.536562 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.536774 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.537016 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.537208 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":900,"threads_started":1}
W20260927 05:12:35.538755 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46442
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:35.634867 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.621361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.97.1:41195)
0927 05:12:35.621593 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.621602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.622135 (+ 533us) server_negotiation.cc:407] Connection header received
0927 05:12:35.622332 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.622336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.622433 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.622515 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.623958 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.624634 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.625838 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.626221 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.626312 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.630340 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.630345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.630348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.630376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.632758 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.633269 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.633272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.633275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.633388 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.633667 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.633670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.633672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.633899 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.634064 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.634288 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.634503 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:35.636442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 501
I20260927 05:12:35.649906 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.637962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.97.1:41195)
0927 05:12:35.638164 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.638170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.638186 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:12:35.638224 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.638228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.638279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.638391 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.639752 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.640583 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.642032 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.642545 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.642686 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.645122 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.645129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.645136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.645173 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.647531 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.648127 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.648134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.648138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.648266 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.648605 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.648611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.648615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.648843 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.649070 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.649345 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.649540 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:12:35.650621 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:35.741146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 501
I20260927 05:12:35.757038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 635
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:12:35.772498 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.757891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.97.2:34469)
0927 05:12:35.758503 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.758510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.758536 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:12:35.758597 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.758603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.758814 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.758948 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.760084 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.761137 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.762295 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.762837 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.762933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.766960 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.766971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.766986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.767101 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.770140 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.770673 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.770680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.770683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.770819 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.771281 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.771285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.771287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.771606 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.771828 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.772011 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.772201 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":317,"threads_started":1}
I20260927 05:12:35.773842 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:35.858245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 635
I20260927 05:12:35.880502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 768
Sep 27 05:12:35 dist-test-slave-1t6p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:12:35.896545 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:35.881434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.97.3:34279)
0927 05:12:35.881888 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:35.881895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:35.881924 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:12:35.881983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:35.881989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:35.882196 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:35.882384 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:35.883679 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.884673 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.886050 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:35.886580 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:35.886722 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:35.890703 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:35.890715 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:35.890729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:35.890762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:35.893316 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.893952 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.893960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.893962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.894075 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:35.894454 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:35.894459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:35.894463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:35.894789 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:35.895926 (+ 1137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:35.896055 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 05:12:35.896247 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":208,"threads_started":1}
I20260927 05:12:35.897797 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:35.981379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 768
I20260927 05:12:35.998265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 408
I20260927 05:12:35.999503 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:36.103195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 408
2026-09-27T05:12:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5023 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:37 dist-test-slave-1t6p krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:37 dist-test-slave-1t6p krb5kdc[918](info): set up 2 sockets
Sep 27 05:12:37 dist-test-slave-1t6p krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260927 05:12:38.325238 388 mini_kdc.cc:121] Time spent starting KDC: real 2.180s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 05:12:38.626031 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.301s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 05:12:38.944590 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.318s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 05:12:39.190318 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.246s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/test-user.keytab.
W20260927 05:12:39.438944 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.248s user 0.000s sys 0.004s
Sep 27 05:12:39 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:39.514187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39653
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:39.821339 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:39.821604 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:39.821688 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:39.828752 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:39.828846 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:39.828890 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:39.828946 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:39.828989 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:39.838234 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:39.839368 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:39.840528 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:39.852250 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:39.852254 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:39.852835 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:39.853122 931 server_base.cc:1034] running on GCE node
I20260927 05:12:39.853797 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:39.855137 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:39.856339 931 hybrid_clock.cc:648] HybridClock initialized: now 1790485959856285 us; error 45 us; skew 500 ppm
I20260927 05:12:39.856693 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:39 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:39.859694 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:39.861053 931 webserver.cc:460] Webserver started at http://127.0.97.62:34927/ using document root <none> and password file <none>
I20260927 05:12:39.861702 931 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:39.861826 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:39.862124 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:39.865003 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "d4cdfd3c15f04faeae2748c99a1e4de5"
format_stamp: "Formatted at 2026-09-27 05:12:39 on dist-test-slave-1t6p"
server_key: "a803654b3af228479cffb0246bf323e3"
server_key_iv: "a2c9824db70fd151a91a7b917c64cf65"
server_key_version: "encryptionkey@0"
I20260927 05:12:39.865746 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "d4cdfd3c15f04faeae2748c99a1e4de5"
format_stamp: "Formatted at 2026-09-27 05:12:39 on dist-test-slave-1t6p"
server_key: "a803654b3af228479cffb0246bf323e3"
server_key_iv: "a2c9824db70fd151a91a7b917c64cf65"
server_key_version: "encryptionkey@0"
I20260927 05:12:39.870594 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:12:39.874776 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:39.876037 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:12:39.876222 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "d4cdfd3c15f04faeae2748c99a1e4de5"
format_stamp: "Formatted at 2026-09-27 05:12:39 on dist-test-slave-1t6p"
server_key: "a803654b3af228479cffb0246bf323e3"
server_key_iv: "a2c9824db70fd151a91a7b917c64cf65"
server_key_version: "encryptionkey@0"
I20260927 05:12:39.876433 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:39.886870 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:39.890635 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:39.890951 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:39.907712 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39653
I20260927 05:12:39.907742 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39653 every 8 connection(s)
I20260927 05:12:39.909046 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:12:39.912601 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:39.918469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 931
I20260927 05:12:39.918684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:12:39.919375 388 external_mini_cluster.cc:1428] Setting key 82294f6110d8026db6d59a0e41d909c9
I20260927 05:12:39.922982 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: Bootstrap starting.
I20260927 05:12:39.926610 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:39.927685 998 log.cc:824] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:39.930202 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: No bootstrap required, opened a new log
Sep 27 05:12:39 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:39.937222 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260927 05:12:39.937565 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:39.937700 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cdfd3c15f04faeae2748c99a1e4de5, State: Initialized, Role: FOLLOWER
I20260927 05:12:39.938318 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260927 05:12:39.938606 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:12:39.938756 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:12:39.938939 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:39.941792 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:39.921324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.0.97.62:39653)
0927 05:12:39.922116 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:39.922131 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:39.922187 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:12:39.924562 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:39.924600 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:39.925088 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:39.925513 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:39.926899 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:39.928304 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:39.929527 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:39.930166 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:39.930355 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:39.934152 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:39.934162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:39.934176 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:39.934208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:39.937260 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:39.938087 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:39.938094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:39.938101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:39.938297 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:39.938651 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:39.938656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:39.938662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:39.939381 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:39.939676 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:39.940111 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 05:12:39.940535 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":350,"threads_started":1}
I20260927 05:12:39.942075 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260927 05:12:39.942591 998 leader_election.cc:302] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4cdfd3c15f04faeae2748c99a1e4de5; no voters:
I20260927 05:12:39.943235 998 leader_election.cc:288] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:12:39.943357 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:39.945684 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:12:39.947297 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [term 1 LEADER]: Becoming Leader. State: Replica: d4cdfd3c15f04faeae2748c99a1e4de5, State: Running, Role: LEADER
I20260927 05:12:39.948017 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260927 05:12:39.951159 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4cdfd3c15f04faeae2748c99a1e4de5. Latest consensus state: current_term: 1 leader_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } } }
I20260927 05:12:39.951565 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:39.952126 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cdfd3c15f04faeae2748c99a1e4de5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } } }
I20260927 05:12:39.952816 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5 [sys.catalog]: This master's current role is: LEADER
W20260927 05:12:39.956508 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:12:39.956712 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:12:39.959452 1018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:12:39.961071 1018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:12:39.968967 1018 catalog_manager.cc:1347] Generated new cluster ID: 172503f5d35941c89f5e2eae304c03b1
I20260927 05:12:39.969072 1018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:12:39.995182 1018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:12:39.996532 1018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:12:40.006007 1018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4cdfd3c15f04faeae2748c99a1e4de5: Generated new TSK 0
I20260927 05:12:40.006795 1018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:12:40.047081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:40.355115 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:40.355448 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:40.355619 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:40.362564 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:40.362701 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:40.362882 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:12:40.372102 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:40.373322 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:40.374593 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:40.387722 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:40.387749 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:40.387912 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:40.388664 1024 server_base.cc:1034] running on GCE node
I20260927 05:12:40.389251 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:40.390067 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:40.391249 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790485960391199 us; error 38 us; skew 500 ppm
I20260927 05:12:40.391534 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:40 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:40.394579 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:12:40.395962 1024 webserver.cc:460] Webserver started at http://127.0.97.1:42581/ using document root <none> and password file <none>
I20260927 05:12:40.396579 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:40.396708 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:40.397007 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:40.399720 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "c894a0347f144bd1a0f78c934ccc8ba1"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1b26d806d4dea5568a1da503c8e29b5b"
server_key_iv: "24117952bdebe81b767b526850d93b4f"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.400444 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c894a0347f144bd1a0f78c934ccc8ba1"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1b26d806d4dea5568a1da503c8e29b5b"
server_key_iv: "24117952bdebe81b767b526850d93b4f"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.405206 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:40.408293 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.409407 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:12:40.409574 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "c894a0347f144bd1a0f78c934ccc8ba1"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1b26d806d4dea5568a1da503c8e29b5b"
server_key_iv: "24117952bdebe81b767b526850d93b4f"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.409765 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:40.441628 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:40.445353 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:40.445644 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:40.446561 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:40.447991 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:40.448093 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.448256 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:40.448326 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.480829 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45783
I20260927 05:12:40.480907 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45783 every 8 connection(s)
I20260927 05:12:40.482095 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:12:40.482542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1024
I20260927 05:12:40.482786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:12:40.483672 388 external_mini_cluster.cc:1428] Setting key 310cf22cfef48f7ca0378f29e2c8b171
Sep 27 05:12:40 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:12:40.503661 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:40.484835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41667 (local address 127.0.97.62:39653)
0927 05:12:40.485568 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:40.485575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:40.486504 (+ 929us) server_negotiation.cc:407] Connection header received
0927 05:12:40.488313 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:40.488321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:40.488406 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:40.488520 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:40.490129 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.490841 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.493395 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.493783 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.493874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:40.498059 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:40.498064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:40.498068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:40.498099 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:40.500747 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:40.501430 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:40.501437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:40.501446 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:40.501556 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:40.501966 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:40.501971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:40.501974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:40.502237 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:40.502389 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:40.503259 (+ 870us) server_negotiation.cc:299] Negotiation successful
0927 05:12:40.503409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":348,"threads_started":1}
I20260927 05:12:40.505173 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:40.485526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.1:41667)
0927 05:12:40.486332 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:40.486377 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:40.488060 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:40.488704 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:40.488712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:40.489196 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:40.489947 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:40.489973 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.490992 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.491022 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:40.493239 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:40.493267 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.494003 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.494012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:40.494341 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:40.495096 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:40.495136 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:40.497843 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:40.500884 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:40.500893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:40.500909 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:40.501304 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:40.501671 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:40.501676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:40.501680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:40.501850 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:40.502383 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:40.502390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:40.502832 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 05:12:40.503155 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 05:12:40.503722 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":353,"threads_started":1}
I20260927 05:12:40.506857 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260927 05:12:40.507210 1152 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:40.508054 1152 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260927 05:12:40.510090 963 ts_manager.cc:194] Registered new tserver with Master: c894a0347f144bd1a0f78c934ccc8ba1 (127.0.97.1:45783)
I20260927 05:12:40.511489 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:12:40.524014 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:40.828361 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:40.828603 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:40.828765 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:40.835438 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:40.835528 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:40.835741 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:12:40.844846 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:40.846127 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:40.847525 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:40.860354 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:40.860332 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:40.860565 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:40.860983 1158 server_base.cc:1034] running on GCE node
I20260927 05:12:40.861590 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:40.862447 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:40.863659 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790485960863623 us; error 28 us; skew 500 ppm
I20260927 05:12:40.863925 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:40 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:40.866830 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:12:40.868136 1158 webserver.cc:460] Webserver started at http://127.0.97.2:44255/ using document root <none> and password file <none>
I20260927 05:12:40.868716 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:40.868813 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:40.869048 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:40.871841 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1531eeb8a198258c0795fb803196c699"
server_key_iv: "e31e13200e6c19b167f3c36bbfb54b4e"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.872524 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1531eeb8a198258c0795fb803196c699"
server_key_iv: "e31e13200e6c19b167f3c36bbfb54b4e"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.878176 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:12:40.881620 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.882920 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:12:40.883139 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb"
format_stamp: "Formatted at 2026-09-27 05:12:40 on dist-test-slave-1t6p"
server_key: "1531eeb8a198258c0795fb803196c699"
server_key_iv: "e31e13200e6c19b167f3c36bbfb54b4e"
server_key_version: "encryptionkey@0"
I20260927 05:12:40.883316 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:40.896026 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:40.899585 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:40.899809 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:40.900576 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:40.901782 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:40.901850 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.901928 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:40.901980 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:40.936478 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39539
I20260927 05:12:40.936518 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39539 every 8 connection(s)
I20260927 05:12:40.937688 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:12:40.938076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1158
I20260927 05:12:40.938248 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:12:40.938879 388 external_mini_cluster.cc:1428] Setting key 3f1bc4928bb20fa62dbfd1aa1bbcecb3
Sep 27 05:12:40 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:12:40.959056 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:40.940367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45941 (local address 127.0.97.62:39653)
0927 05:12:40.940564 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:40.940570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:40.941833 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 05:12:40.943571 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:40.943576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:40.943651 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:40.943768 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:40.945409 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.946646 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.948216 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.948991 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.949127 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:40.952921 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:40.952928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:40.952934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:40.952982 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:40.956126 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:40.956831 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:40.956835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:40.956837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:40.956996 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:40.957443 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:40.957447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:40.957449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:40.957708 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:40.957813 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:40.958610 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 05:12:40.958782 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:40.960654 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:40.940960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.2:45941)
0927 05:12:40.941690 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:40.941729 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:40.943339 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:40.944005 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:40.944017 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:40.944702 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:40.945256 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:40.945271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.946793 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.946797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:40.948079 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:40.948091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:40.949201 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:40.949209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:40.949403 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:40.950161 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:40.950182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:40.952678 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:40.956289 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:40.956299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:40.956323 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:40.956716 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:40.957099 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:40.957102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:40.957104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:40.957332 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:40.957829 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:40.957839 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:40.958215 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 05:12:40.958521 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 05:12:40.959127 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":331,"threads_started":1}
I20260927 05:12:40.962042 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260927 05:12:40.962383 1286 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:40.963155 1286 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260927 05:12:40.964555 963 ts_manager.cc:194] Registered new tserver with Master: 6a2f4fd5f1bb4bb3996a7b8eafe148bb (127.0.97.2:39539)
I20260927 05:12:40.965798 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:12:40.980047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:41.284062 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:41.284292 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:41.284448 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:41.292037 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:41.292146 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:41.292315 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:12:41.301486 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:41.302579 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:41.303840 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:41.316259 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:41.316359 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:41.316334 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:41.317056 1291 server_base.cc:1034] running on GCE node
I20260927 05:12:41.317613 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:41.318413 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:41.319617 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790485961319585 us; error 47 us; skew 500 ppm
I20260927 05:12:41.319895 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:41.322777 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:12:41.324136 1291 webserver.cc:460] Webserver started at http://127.0.97.3:44329/ using document root <none> and password file <none>
I20260927 05:12:41.324708 1291 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:41.324803 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:41.325042 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:41.327726 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "33614c3914e941eda840867a4c08c837"
format_stamp: "Formatted at 2026-09-27 05:12:41 on dist-test-slave-1t6p"
server_key: "57f2b3804e8d621e0cb5cdbf8b620ed1"
server_key_iv: "b9fb02c11d13209c8823b6199594da3d"
server_key_version: "encryptionkey@0"
I20260927 05:12:41.328444 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "33614c3914e941eda840867a4c08c837"
format_stamp: "Formatted at 2026-09-27 05:12:41 on dist-test-slave-1t6p"
server_key: "57f2b3804e8d621e0cb5cdbf8b620ed1"
server_key_iv: "b9fb02c11d13209c8823b6199594da3d"
server_key_version: "encryptionkey@0"
I20260927 05:12:41.333050 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 05:12:41.336021 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:41.337277 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:12:41.337396 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "33614c3914e941eda840867a4c08c837"
format_stamp: "Formatted at 2026-09-27 05:12:41 on dist-test-slave-1t6p"
server_key: "57f2b3804e8d621e0cb5cdbf8b620ed1"
server_key_iv: "b9fb02c11d13209c8823b6199594da3d"
server_key_version: "encryptionkey@0"
I20260927 05:12:41.337548 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:41.370496 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:41.374145 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:41.374374 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:41.375350 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:41.376574 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:41.376688 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:41.376843 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:41.376916 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:41.409178 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37175
I20260927 05:12:41.409220 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37175 every 8 connection(s)
I20260927 05:12:41.410415 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:12:41.414724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1291
I20260927 05:12:41.414899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:12:41.415472 388 external_mini_cluster.cc:1428] Setting key 7dd899aa64a74834269fe795a14824fb
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:41.429638 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.412836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33181 (local address 127.0.97.62:39653)
0927 05:12:41.413025 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.413030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.414132 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 05:12:41.416502 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.416507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.416595 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.416742 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.418494 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.419282 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.420701 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.421084 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.421179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.424189 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.424193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.424197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.424225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.426911 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.427554 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.427558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.427560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.427663 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.428021 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.428023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.428025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.428261 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.428362 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.429094 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.429296 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 05:12:41.430630 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.413309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.3:33181)
0927 05:12:41.413973 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.414008 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.416273 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.417025 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.417037 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.417687 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:41.418309 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.418328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.419418 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.419421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.420582 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.420593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.421257 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.421261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.421442 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.422015 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:41.422038 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:41.424002 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:41.427070 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:41.427076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:41.427087 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:41.427460 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:41.427770 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:41.427774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:41.427776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:41.427929 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:41.428356 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:41.428361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:41.428666 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.428956 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.429415 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":272,"threads_started":1}
I20260927 05:12:41.431845 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260927 05:12:41.432113 1419 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:41.432745 1419 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260927 05:12:41.434052 963 ts_manager.cc:194] Registered new tserver with Master: 33614c3914e941eda840867a4c08c837 (127.0.97.3:37175)
I20260927 05:12:41.434801 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33181
I20260927 05:12:41.442335 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:12:41.460371 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.448260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.0.97.62:39653)
0927 05:12:41.448398 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.448402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.448880 (+ 478us) server_negotiation.cc:407] Connection header received
0927 05:12:41.449019 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.449022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.449082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.449163 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.450573 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.451429 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.453062 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.453482 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.453574 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.455789 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.455794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.455797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.455826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.458084 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.458638 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.458641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.458644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.458762 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.459149 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.459152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.459154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.459419 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.459589 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.459863 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.460014 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 05:12:41.466639 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:41.473623 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:41.495635 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.485098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45783 (local address 127.0.0.1:44860)
0927 05:12:41.488172 (+ 3074us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.488190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.488304 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.488922 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.488933 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.489244 (+ 311us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:41.489709 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.489726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.491223 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.491231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.493249 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.493260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.494610 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.494616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.494805 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.494853 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.494982 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.495190 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2969,"mutex_wait_us":31,"spinlock_wait_cycles":614400,"thread_start_us":481,"threads_started":1}
I20260927 05:12:41.496909 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.485474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.0.97.1:45783)
0927 05:12:41.487900 (+ 2426us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.487906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.488378 (+ 472us) server_negotiation.cc:407] Connection header received
0927 05:12:41.488480 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.488486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.488718 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.489129 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:41.490022 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.491101 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.493393 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.494393 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.494558 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.494781 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.495054 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.496220 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2295,"thread_start_us":2168,"threads_started":1}
I20260927 05:12:41.497094 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.484909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37175 (local address 127.0.0.1:47304)
0927 05:12:41.485944 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.485986 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.486175 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.489192 (+ 3017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.489200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.489231 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:41.489906 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.489918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.491843 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.491848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.494098 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.494111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.495821 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.495827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.496000 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.496032 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.496107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.496228 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":156,"spinlock_wait_cycles":2304,"thread_start_us":1619,"threads_started":1}
I20260927 05:12:41.497089 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.486972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.97.2:39539)
0927 05:12:41.488289 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.488301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.489345 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 05:12:41.489438 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.489446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.489656 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.489835 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:41.490540 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.491987 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.494464 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.495873 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.495981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.496154 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.496632 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.496786 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":1017,"threads_started":1}
I20260927 05:12:41.497200 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.486509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39539 (local address 127.0.0.1:36496)
0927 05:12:41.488657 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.488676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.489220 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.489989 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.489992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.490038 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:41.490361 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.490375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.492139 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.492144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.494334 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.494343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.496312 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.496318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.496467 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.496488 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.496584 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.496698 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"mutex_wait_us":13,"thread_start_us":1891,"threads_started":1}
I20260927 05:12:41.498368 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.485604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.0.97.3:37175)
0927 05:12:41.488182 (+ 2578us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.488192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.488224 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:12:41.488325 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.488335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.488581 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.489559 (+ 978us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:41.490101 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.491697 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.494259 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.495604 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.497688 (+ 2084us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.497890 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.497952 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.498104 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":1795,"threads_started":1}
I20260927 05:12:41.500850 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 06467dd8dde8457a875b2dd06c926326 (DEFAULT_TABLE table=test-table [id=b8d232bf030b43bcb606e00d6193497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:41.500898 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 06467dd8dde8457a875b2dd06c926326 (DEFAULT_TABLE table=test-table [id=b8d232bf030b43bcb606e00d6193497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:41.501485 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 06467dd8dde8457a875b2dd06c926326 (DEFAULT_TABLE table=test-table [id=b8d232bf030b43bcb606e00d6193497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:41.502038 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06467dd8dde8457a875b2dd06c926326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:41.502171 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06467dd8dde8457a875b2dd06c926326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:41.502671 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06467dd8dde8457a875b2dd06c926326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:41.513067 1437 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Bootstrap starting.
I20260927 05:12:41.515213 1152 heartbeater.cc:503] Master 127.0.97.62:39653 was elected leader, sending a full tablet report...
I20260927 05:12:41.517230 1437 tablet_bootstrap.cc:654] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:41.518703 1437 log.cc:824] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:41.519336 1438 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Bootstrap starting.
I20260927 05:12:41.523375 1438 tablet_bootstrap.cc:654] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:41.523672 1439 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Bootstrap starting.
I20260927 05:12:41.524868 1438 log.cc:824] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:41.527033 1439 tablet_bootstrap.cc:654] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:41.527935 1439 log.cc:824] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:41.528198 1437 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: No bootstrap required, opened a new log
I20260927 05:12:41.528191 1438 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: No bootstrap required, opened a new log
I20260927 05:12:41.528437 1438 ts_tablet_manager.cc:1397] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:12:41.528437 1437 ts_tablet_manager.cc:1397] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260927 05:12:41.532894 1439 tablet_bootstrap.cc:492] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: No bootstrap required, opened a new log
I20260927 05:12:41.533138 1439 ts_tablet_manager.cc:1397] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 05:12:41.539134 1438 raft_consensus.cc:348] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.539140 1437 raft_consensus.cc:348] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.539541 1437 raft_consensus.cc:374] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:41.539541 1438 raft_consensus.cc:374] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:41.539644 1438 raft_consensus.cc:729] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c894a0347f144bd1a0f78c934ccc8ba1, State: Initialized, Role: FOLLOWER
I20260927 05:12:41.539680 1437 raft_consensus.cc:729] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2f4fd5f1bb4bb3996a7b8eafe148bb, State: Initialized, Role: FOLLOWER
I20260927 05:12:41.540140 1439 raft_consensus.cc:348] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.540501 1439 raft_consensus.cc:374] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:41.540599 1439 raft_consensus.cc:729] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33614c3914e941eda840867a4c08c837, State: Initialized, Role: FOLLOWER
I20260927 05:12:41.540699 1437 consensus_queue.cc:260] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.540704 1438 consensus_queue.cc:260] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.541208 1439 consensus_queue.cc:260] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.543844 1438 ts_tablet_manager.cc:1428] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260927 05:12:41.547801 1286 heartbeater.cc:503] Master 127.0.97.62:39653 was elected leader, sending a full tablet report...
I20260927 05:12:41.548126 1419 heartbeater.cc:503] Master 127.0.97.62:39653 was elected leader, sending a full tablet report...
I20260927 05:12:41.548252 1437 ts_tablet_manager.cc:1428] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260927 05:12:41.549499 1439 ts_tablet_manager.cc:1428] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260927 05:12:41.585688 1444 raft_consensus.cc:482] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:41.585983 1444 raft_consensus.cc:504] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.587873 1444 leader_election.cc:288] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33614c3914e941eda840867a4c08c837 (127.0.97.3:37175), c894a0347f144bd1a0f78c934ccc8ba1 (127.0.97.1:45783)
I20260927 05:12:41.594769 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.588184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38435 (local address 127.0.97.3:37175)
0927 05:12:41.588405 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.588414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.589129 (+ 715us) server_negotiation.cc:407] Connection header received
0927 05:12:41.589264 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.589269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.589345 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.589430 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:41.590440 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.591295 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.593041 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.593796 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.593888 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.593983 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.594381 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.594507 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:41.594817 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.588353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44143 (local address 127.0.97.1:45783)
0927 05:12:41.588565 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.588573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.589298 (+ 725us) server_negotiation.cc:407] Connection header received
0927 05:12:41.589433 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.589438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.589510 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.589609 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:41.590655 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.591435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.593106 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.593881 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.594017 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.594110 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.594465 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.594589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:12:41.594857 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.588021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37175 (local address 127.0.97.2:38435)
0927 05:12:41.589029 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.589044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.589156 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.589461 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.589466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.589498 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:41.590296 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.590306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.591419 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.591424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.592897 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.592908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.594007 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.594014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.594231 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.594260 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.594331 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.594445 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":25,"thread_start_us":653,"threads_started":1}
I20260927 05:12:41.595064 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.588264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45783 (local address 127.0.97.2:44143)
0927 05:12:41.589161 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:41.589177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:41.589296 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:41.589639 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:41.589642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:41.589716 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:41.590501 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.590513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.591576 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.591579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.592938 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:41.592948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.594101 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.594105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:41.594288 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.594331 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 05:12:41.594421 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 05:12:41.594530 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":36,"spinlock_wait_cycles":46464,"thread_start_us":554,"threads_started":1}
I20260927 05:12:41.595865 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06467dd8dde8457a875b2dd06c926326" candidate_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33614c3914e941eda840867a4c08c837" is_pre_election: true
I20260927 05:12:41.596009 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06467dd8dde8457a875b2dd06c926326" candidate_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" is_pre_election: true
I20260927 05:12:41.596362 1374 raft_consensus.cc:2454] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f4fd5f1bb4bb3996a7b8eafe148bb in term 0.
I20260927 05:12:41.596520 1107 raft_consensus.cc:2454] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f4fd5f1bb4bb3996a7b8eafe148bb in term 0.
I20260927 05:12:41.596974 1177 leader_election.cc:302] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33614c3914e941eda840867a4c08c837, 6a2f4fd5f1bb4bb3996a7b8eafe148bb; no voters:
I20260927 05:12:41.597337 1444 raft_consensus.cc:2781] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:41.597478 1444 raft_consensus.cc:482] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:41.597599 1444 raft_consensus.cc:3037] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:41.599781 1444 raft_consensus.cc:504] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.600373 1444 leader_election.cc:288] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [CANDIDATE]: Term 1 election: Requested vote from peers 33614c3914e941eda840867a4c08c837 (127.0.97.3:37175), c894a0347f144bd1a0f78c934ccc8ba1 (127.0.97.1:45783)
I20260927 05:12:41.600769 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06467dd8dde8457a875b2dd06c926326" candidate_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33614c3914e941eda840867a4c08c837"
I20260927 05:12:41.600795 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06467dd8dde8457a875b2dd06c926326" candidate_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c894a0347f144bd1a0f78c934ccc8ba1"
I20260927 05:12:41.600972 1374 raft_consensus.cc:3037] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:41.600971 1107 raft_consensus.cc:3037] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:41.604490 1374 raft_consensus.cc:2454] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f4fd5f1bb4bb3996a7b8eafe148bb in term 1.
I20260927 05:12:41.604490 1107 raft_consensus.cc:2454] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f4fd5f1bb4bb3996a7b8eafe148bb in term 1.
I20260927 05:12:41.604902 1175 leader_election.cc:302] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f4fd5f1bb4bb3996a7b8eafe148bb, c894a0347f144bd1a0f78c934ccc8ba1; no voters:
I20260927 05:12:41.605189 1444 raft_consensus.cc:2781] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:41.605531 1444 raft_consensus.cc:686] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [term 1 LEADER]: Becoming Leader. State: Replica: 6a2f4fd5f1bb4bb3996a7b8eafe148bb, State: Running, Role: LEADER
I20260927 05:12:41.605984 1444 consensus_queue.cc:237] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } }
I20260927 05:12:41.610352 961 catalog_manager.cc:5495] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a2f4fd5f1bb4bb3996a7b8eafe148bb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33614c3914e941eda840867a4c08c837" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c894a0347f144bd1a0f78c934ccc8ba1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2f4fd5f1bb4bb3996a7b8eafe148bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39539 } health_report { overall_health: HEALTHY } } }
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:41.656437 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.641206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.0.97.1:45783)
0927 05:12:41.641430 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.641440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.641918 (+ 478us) server_negotiation.cc:407] Connection header received
0927 05:12:41.642087 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.642090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.642167 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.642266 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.643711 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.644974 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.646300 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.646720 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.646853 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.651107 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.651119 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.651134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.651165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.653979 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.654572 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.654579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.654581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.654694 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.655063 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.655066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.655068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.655392 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.655715 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.655926 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.656071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 05:12:41.662037 1420 tablet.cc:2369] T 06467dd8dde8457a875b2dd06c926326 P 33614c3914e941eda840867a4c08c837: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 05:12:41.689672 1287 tablet.cc:2369] T 06467dd8dde8457a875b2dd06c926326 P 6a2f4fd5f1bb4bb3996a7b8eafe148bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:41.702206 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.689136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.97.1:45783)
0927 05:12:41.689338 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.689345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.689568 (+ 223us) server_negotiation.cc:407] Connection header received
0927 05:12:41.689764 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.689767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.689827 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.689947 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.691792 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.692501 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.693632 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.694004 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.694090 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.697619 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.697624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.697627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.697655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.700188 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.700741 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.700745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.700747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.700866 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.701221 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.701224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.701226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.701458 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.701627 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.701842 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.701984 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 05:12:41.734508 1153 tablet.cc:2369] T 06467dd8dde8457a875b2dd06c926326 P c894a0347f144bd1a0f78c934ccc8ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:41.742792 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.729331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.97.1:45783)
0927 05:12:41.729541 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.729548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.729837 (+ 289us) server_negotiation.cc:407] Connection header received
0927 05:12:41.730042 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.730046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.730103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.730204 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.731536 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.732224 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.733434 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.733853 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.733984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.737999 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.738003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.738006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.738034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.740534 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.741122 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.741129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.741132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.741267 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.741719 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.741723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.741725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.741982 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.742161 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.742391 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.742548 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 05:12:41.744565 1067 tablet_service.cc:2908] Rejecting scan request for tablet 06467dd8dde8457a875b2dd06c926326: Uninitialized: clean time has not yet been initialized
I20260927 05:12:41.747649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1024
I20260927 05:12:41.759938 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.748805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.0.97.1:45783)
0927 05:12:41.748945 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.748949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.749336 (+ 387us) server_negotiation.cc:407] Connection header received
0927 05:12:41.749524 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.749527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.749589 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.749667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.750965 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.751721 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.752835 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.753233 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.753333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.755297 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.755302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.755305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.755331 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.757783 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.758338 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.758341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.758344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.758449 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.758783 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.758787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.758789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.759121 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.759334 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.759545 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.759720 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:12:41.760545 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:41.847319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1024
I20260927 05:12:41.864141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1158
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:12:41.883626 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.864839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.0.97.2:39539)
0927 05:12:41.865036 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.865042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.865060 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:12:41.865265 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.865270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.865331 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.865441 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.866564 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.867330 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.868598 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.868996 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.869127 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.877012 (+ 7885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.877025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.877040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.877071 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:41.881001 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.881820 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.881827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.881830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.881961 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:41.882275 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:41.882278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:41.882280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:41.882586 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:41.882807 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:41.882982 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 05:12:41.883286 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:41.884277 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:41.972528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1158
I20260927 05:12:41.988931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1291
Sep 27 05:12:41 dist-test-slave-1t6p krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:12:42.003487 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:41.989827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.0.97.3:37175)
0927 05:12:41.989999 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:41.990004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:41.990024 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:12:41.990152 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:41.990157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:41.990223 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:41.990311 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:41.991589 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.992314 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.993475 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:41.993842 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:41.993931 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:41.997948 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:41.997960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:41.997976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:41.998011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:42.000783 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:42.001433 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:42.001440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:42.001442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:42.001575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:42.001934 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:42.001941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:42.001944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:42.002281 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:42.002585 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:42.002827 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 05:12:42.003189 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:42.004073 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:42.088900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1291
I20260927 05:12:42.104566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 931
I20260927 05:12:42.105684 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:42.211290 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 931
2026-09-27T05:12:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6108 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:42 dist-test-slave-1t6p krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:42 dist-test-slave-1t6p krb5kdc[1467](info): set up 2 sockets
Sep 27 05:12:42 dist-test-slave-1t6p krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260927 05:12:43.290957 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:12:43 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:43.458007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39511
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:43.769919 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:43.770177 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:43.770373 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:43.777410 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:43.777500 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:12:43.777592 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:43.777670 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:43.777760 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:43.777858 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:43.787350 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:43.788555 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:43.789819 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:43.801352 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:43.801533 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:43.801506 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:43.802446 1480 server_base.cc:1034] running on GCE node
I20260927 05:12:43.803166 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:43.804430 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:43.805657 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790485963805599 us; error 33 us; skew 500 ppm
I20260927 05:12:43.805976 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:43 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:43.808911 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:43.810199 1480 webserver.cc:460] Webserver started at http://127.0.97.62:45943/ using document root <none> and password file <none>
I20260927 05:12:43.810842 1480 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:43.810951 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:43.811337 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:43.814090 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6"
format_stamp: "Formatted at 2026-09-27 05:12:43 on dist-test-slave-1t6p"
server_key: "67ca38cabad863a1e960aa5fdc562607"
server_key_iv: "7a14f6f169c5564f927e2b551d84676e"
server_key_version: "encryptionkey@0"
I20260927 05:12:43.814894 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6"
format_stamp: "Formatted at 2026-09-27 05:12:43 on dist-test-slave-1t6p"
server_key: "67ca38cabad863a1e960aa5fdc562607"
server_key_iv: "7a14f6f169c5564f927e2b551d84676e"
server_key_version: "encryptionkey@0"
I20260927 05:12:43.819693 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:12:43.823052 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:43.824299 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:43.824465 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6"
format_stamp: "Formatted at 2026-09-27 05:12:43 on dist-test-slave-1t6p"
server_key: "67ca38cabad863a1e960aa5fdc562607"
server_key_iv: "7a14f6f169c5564f927e2b551d84676e"
server_key_version: "encryptionkey@0"
I20260927 05:12:43.824662 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:43.864429 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:43.868580 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:43.868858 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:43.890357 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39511
I20260927 05:12:43.890380 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39511 every 8 connection(s)
I20260927 05:12:43.891801 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:12:43.892985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1480
I20260927 05:12:43.893194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:12:43.893834 388 external_mini_cluster.cc:1428] Setting key 4de012e090f2498bc34a8075f67c0c2d
I20260927 05:12:43.896154 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:12:43 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:43.908747 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Bootstrap starting.
I20260927 05:12:43.912730 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:43.913893 1557 log.cc:824] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:43.916872 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: No bootstrap required, opened a new log
I20260927 05:12:43.917003 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:43.895727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.0.97.62:39511)
0927 05:12:43.896501 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:43.896513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:43.896552 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:12:43.898124 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:43.898147 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:43.898519 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:43.898849 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:43.900561 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:43.901843 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:43.903562 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:43.904191 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:43.904337 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:43.908935 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:43.908945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:43.908962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:43.909005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:43.912825 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:43.913473 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:43.913481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:43.913488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:43.913705 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:43.914084 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:43.914094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:43.914096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:43.914603 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:43.915243 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:43.915652 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 05:12:43.916013 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":379,"threads_started":1}
I20260927 05:12:43.924252 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } }
I20260927 05:12:43.924609 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:43.924711 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3e4bde0ddd41f78701d78e9dbf5ab6, State: Initialized, Role: FOLLOWER
I20260927 05:12:43.925274 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } }
I20260927 05:12:43.925510 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:12:43.925678 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:12:43.925859 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:43.927963 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } }
I20260927 05:12:43.928448 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e3e4bde0ddd41f78701d78e9dbf5ab6; no voters:
I20260927 05:12:43.929069 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:12:43.929163 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:43.929899 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3e4bde0ddd41f78701d78e9dbf5ab6, State: Running, Role: LEADER
I20260927 05:12:43.930408 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } }
I20260927 05:12:43.931242 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:12:43.932991 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } } }
I20260927 05:12:43.933291 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:43.934857 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:12:43.934737 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e3e4bde0ddd41f78701d78e9dbf5ab6. Latest consensus state: current_term: 1 leader_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3e4bde0ddd41f78701d78e9dbf5ab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39511 } } }
I20260927 05:12:43.935060 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:43.939536 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:12:43.948797 1568 catalog_manager.cc:1347] Generated new cluster ID: 9a7b7a2c83ab4e4c99ee09ae6b2384ac
I20260927 05:12:43.948894 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:12:43.958849 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:12:43.960153 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:12:43.968863 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Generated new TSK 0
I20260927 05:12:43.969604 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:12:44.015995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39511
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:44.325379 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:44.325677 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:44.325846 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:44.333103 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:44.333235 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:44.333369 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:44.333508 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:12:44.343072 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:44.344276 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:44.345528 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:44.357841 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:44.357877 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:44.357795 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:44.358701 1583 server_base.cc:1034] running on GCE node
I20260927 05:12:44.359368 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:44.360235 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:44.361440 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790485964361381 us; error 33 us; skew 500 ppm
I20260927 05:12:44.361687 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:44 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:44.364678 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:12:44.366006 1583 webserver.cc:460] Webserver started at http://127.0.97.1:39755/ using document root <none> and password file <none>
I20260927 05:12:44.366707 1583 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:44.366832 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:44.367200 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:44.369889 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "8ad9286535934642852fae90a56c9683"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "f2bd7fda7db32f12571cc38c11418174"
server_key_iv: "dcb120298067a69f041acafa49ee2e1a"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.370649 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8ad9286535934642852fae90a56c9683"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "f2bd7fda7db32f12571cc38c11418174"
server_key_iv: "dcb120298067a69f041acafa49ee2e1a"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.376468 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 05:12:44.379640 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:44.380805 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 05:12:44.380968 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "8ad9286535934642852fae90a56c9683"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "f2bd7fda7db32f12571cc38c11418174"
server_key_iv: "dcb120298067a69f041acafa49ee2e1a"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.381172 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:44.393688 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:44.397193 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:44.397429 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:44.399863 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:44.399974 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:44.400107 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:44.400188 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:44 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:44.422073 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.400648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.1:43693)
0927 05:12:44.401525 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:44.401573 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:44.403372 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:44.404105 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:44.404115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:44.404820 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:44.405611 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.405631 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.407429 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.407432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.408633 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.408643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.409796 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.409801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.409960 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.410646 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:44.410698 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:44.414118 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:44.417435 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.417447 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.417466 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.418167 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.418520 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.418528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.418534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.418820 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.419364 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:44.419371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:44.419746 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 05:12:44.420102 (+ 356us) client_negotiation.cc:240] Negotiation successful
0927 05:12:44.420626 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":393,"threads_started":1}
I20260927 05:12:44.422796 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.400207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43693 (local address 127.0.97.62:39511)
0927 05:12:44.400351 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:44.400355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:44.401700 (+ 1345us) server_negotiation.cc:407] Connection header received
0927 05:12:44.403635 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:44.403640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:44.403723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:44.403852 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:44.405799 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.407240 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.408779 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.409549 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.409708 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.414367 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:44.414376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:44.414382 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:44.414423 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:44.417298 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.418287 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.418291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.418293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.418414 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.418927 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.418930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.418932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.419233 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:44.419385 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:44.422202 (+ 2817us) server_negotiation.cc:299] Negotiation successful
0927 05:12:44.422444 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 05:12:44.449750 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38849
I20260927 05:12:44.449772 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38849 every 8 connection(s)
I20260927 05:12:44.451279 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:12:44.453225 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.437826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.1:39555)
0927 05:12:44.438506 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:44.438530 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:44.438697 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:44.439331 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:44.439338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:44.440098 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:44.440478 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.440489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.441488 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.441493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.443787 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.443810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.444808 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.444817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.445056 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.445727 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:44.445756 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:44.447468 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:44.450537 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.450542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.450545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.451088 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.451435 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.451440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.451444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.451581 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.452157 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:44.452163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:44.452386 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 05:12:44.452547 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 05:12:44.452881 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":384,"threads_started":1}
I20260927 05:12:44.453584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1583
I20260927 05:12:44.453401 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39511
I20260927 05:12:44.453788 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:12:44.453828 1718 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:44.454218 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.437733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39555 (local address 127.0.97.62:39511)
0927 05:12:44.437914 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:44.437921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:44.438638 (+ 717us) server_negotiation.cc:407] Connection header received
0927 05:12:44.438902 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:44.438908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:44.439006 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:44.439140 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:44.440616 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.441342 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.444017 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.444588 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.444736 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.447673 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:44.447683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:44.447688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:44.447731 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:44.450411 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.451194 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.451197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.451200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.451320 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.451695 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.451707 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.451709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.451971 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:44.453640 (+ 1669us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:44.453786 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 05:12:44.453997 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:12:44.454612 388 external_mini_cluster.cc:1428] Setting key d89755f0579905387d36e9a63b6bab5e
I20260927 05:12:44.454639 1718 heartbeater.cc:511] Master 127.0.97.62:39511 requested a full tablet report, sending...
I20260927 05:12:44.456898 1511 ts_manager.cc:194] Registered new tserver with Master: 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849)
I20260927 05:12:44.458518 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43693
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:12:44.496241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39511
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:44.802795 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:44.803058 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:44.803179 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:44.810151 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:44.810272 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:44.810429 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:44.810542 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:12:44.820631 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:44.821816 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:44.823448 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:44.836587 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:44.836769 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:44.836612 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:44.837386 1722 server_base.cc:1034] running on GCE node
I20260927 05:12:44.837996 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:44.838835 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:44.840073 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790485964840001 us; error 44 us; skew 500 ppm
I20260927 05:12:44.840389 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:44 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:44.848558 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:12:44.849880 1722 webserver.cc:460] Webserver started at http://127.0.97.2:45099/ using document root <none> and password file <none>
I20260927 05:12:44.850577 1722 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:44.850662 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:44.850919 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:44.853735 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "f8f3478ba0b84446919470f9b95c09a9"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "b11aa5b4b562ff8b2aea874ad25dce23"
server_key_iv: "589f9032140d9dc9346e01cc9ac45b2c"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.854447 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f8f3478ba0b84446919470f9b95c09a9"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "b11aa5b4b562ff8b2aea874ad25dce23"
server_key_iv: "589f9032140d9dc9346e01cc9ac45b2c"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.859517 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:12:44.862862 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:44.864248 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:12:44.864524 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f8f3478ba0b84446919470f9b95c09a9"
format_stamp: "Formatted at 2026-09-27 05:12:44 on dist-test-slave-1t6p"
server_key: "b11aa5b4b562ff8b2aea874ad25dce23"
server_key_iv: "589f9032140d9dc9346e01cc9ac45b2c"
server_key_version: "encryptionkey@0"
I20260927 05:12:44.864741 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:44.881968 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:44.886206 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:44.886471 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:44.888965 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:44.889078 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:44.889217 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:44.889285 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:44 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:44.916272 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.889672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57285 (local address 127.0.97.62:39511)
0927 05:12:44.889858 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:44.889864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:44.893998 (+ 4134us) server_negotiation.cc:407] Connection header received
0927 05:12:44.897012 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:44.897018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:44.897099 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:44.897167 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:44.899417 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.900561 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.903215 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.903857 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.903941 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.908928 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:44.908935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:44.908941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:44.908976 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:44.912441 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.913364 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.913370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.913374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.913500 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.914055 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.914060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.914064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.914387 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:44.914507 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:44.915677 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0927 05:12:44.915915 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:12:44.918605 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.890462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.2:57285)
0927 05:12:44.893839 (+ 3377us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:44.893891 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:44.896650 (+ 2759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:44.897685 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:44.897695 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:44.898415 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:44.899233 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.899255 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.900795 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.900802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.903057 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.903070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.904261 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.904267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.904532 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.905378 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:44.905406 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:44.908667 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:44.912603 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.912611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.912626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.913232 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.913654 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.913659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.913663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.913935 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.914598 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:44.914605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:44.915160 (+ 555us) client_negotiation.cc:769] Sending connection context
0927 05:12:44.916553 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0927 05:12:44.916988 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3108,"thread_start_us":2871,"threads_started":1}
I20260927 05:12:44.938697 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.925183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56041 (local address 127.0.97.62:39511)
0927 05:12:44.925419 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:44.925424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:44.926193 (+ 769us) server_negotiation.cc:407] Connection header received
0927 05:12:44.926379 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:44.926384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:44.926450 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:44.926546 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:44.927514 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.928296 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.929631 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.930046 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.930172 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.932498 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:44.932506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:44.932515 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:44.932542 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:44.935949 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.936786 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.936794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.936798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.936919 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:44.937286 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:44.937291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:44.937295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:44.937587 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:44.937737 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:44.938124 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 05:12:44.938335 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 05:12:44.938813 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:44.925434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.2:56041)
0927 05:12:44.926093 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:44.926112 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:44.926258 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:44.926567 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:44.926571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:44.926998 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:44.927383 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.927400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.928419 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.928425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.929497 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:44.929508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:44.930234 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:44.930240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:44.930393 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:44.931102 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:44.931148 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:44.932309 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:44.936093 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.936098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.936102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.936647 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.937035 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:44.937040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:44.937044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:44.937173 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:44.937702 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:44.937707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:44.937880 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 05:12:44.938031 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 05:12:44.938329 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":415,"threads_started":1}
I20260927 05:12:44.944214 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41385
I20260927 05:12:44.944242 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41385 every 8 connection(s)
I20260927 05:12:44.945360 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:12:44.947290 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39511
I20260927 05:12:44.947511 1857 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:44.948143 1857 heartbeater.cc:511] Master 127.0.97.62:39511 requested a full tablet report, sending...
I20260927 05:12:44.949345 1512 ts_manager.cc:194] Registered new tserver with Master: f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385)
I20260927 05:12:44.950161 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57285
I20260927 05:12:44.951817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1722
I20260927 05:12:44.951982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:12:44.952585 388 external_mini_cluster.cc:1428] Setting key 9b308f9e9f48d5a100c0ad60f877e409
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:12:44.995055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39511
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:45.310050 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:45.310308 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:45.310477 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:45.317411 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:45.317498 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:45.317647 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:45.317754 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:12:45.327355 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:45.328616 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:45.329948 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:45.342458 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:45.342521 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:45.342545 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:45.343230 1861 server_base.cc:1034] running on GCE node
I20260927 05:12:45.343780 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:45.344611 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:45.345845 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790485965345796 us; error 44 us; skew 500 ppm
I20260927 05:12:45.346119 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:45 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:45.349071 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:12:45.350520 1861 webserver.cc:460] Webserver started at http://127.0.97.3:43397/ using document root <none> and password file <none>
I20260927 05:12:45.351264 1861 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:45.351397 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:45.351719 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:45.354470 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "386f7ce2e5d347d0a28f752718462b71"
format_stamp: "Formatted at 2026-09-27 05:12:45 on dist-test-slave-1t6p"
server_key: "4bde22d4ff0cf81e795e7a997e85e550"
server_key_iv: "30ba5984d71a5cbe4f17826a2fa9f826"
server_key_version: "encryptionkey@0"
I20260927 05:12:45.355322 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "386f7ce2e5d347d0a28f752718462b71"
format_stamp: "Formatted at 2026-09-27 05:12:45 on dist-test-slave-1t6p"
server_key: "4bde22d4ff0cf81e795e7a997e85e550"
server_key_iv: "30ba5984d71a5cbe4f17826a2fa9f826"
server_key_version: "encryptionkey@0"
I20260927 05:12:45.360092 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:12:45.363373 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:45.364696 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:12:45.364889 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "386f7ce2e5d347d0a28f752718462b71"
format_stamp: "Formatted at 2026-09-27 05:12:45 on dist-test-slave-1t6p"
server_key: "4bde22d4ff0cf81e795e7a997e85e550"
server_key_iv: "30ba5984d71a5cbe4f17826a2fa9f826"
server_key_version: "encryptionkey@0"
I20260927 05:12:45.365100 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:45.379657 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:45.383131 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:45.383376 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:45.386029 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:45.386125 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:45.386302 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:45.386379 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:45 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:45.409096 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.386494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45879 (local address 127.0.97.62:39511)
0927 05:12:45.386720 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.386730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.388037 (+ 1307us) server_negotiation.cc:407] Connection header received
0927 05:12:45.390049 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.390054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.390143 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.390267 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:45.392454 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.393414 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.396005 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.396699 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.396873 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.400938 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:45.400947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:45.400955 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:45.401007 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:45.405273 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.406234 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.406241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.406245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.406419 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.407084 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.407090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.407096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.407380 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:45.407534 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.408552 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.408765 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:12:45.410050 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.387088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.3:45879)
0927 05:12:45.387883 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.387916 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.389712 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.390514 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.390523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.391357 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:45.392277 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.392297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.393605 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.393610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.395840 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.395861 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.396971 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.396980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.397251 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.397948 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:45.397970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:45.400654 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:45.405438 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.405444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.405454 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.406097 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.406532 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.406539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.406544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.406891 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.407518 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:45.407525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:45.407955 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.408244 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.408749 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":358,"threads_started":1}
I20260927 05:12:45.428731 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.415246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.97.3:60471)
0927 05:12:45.416082 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.416106 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.416261 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.416653 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.416658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.417327 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:45.417846 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.417861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.419184 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.419187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.420460 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.420471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.421154 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.421160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.421314 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.421847 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:45.421862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:45.422446 (+ 584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:45.426064 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.426071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.426078 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.426630 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.427163 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.427173 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.427176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.427296 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.427821 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:45.427825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:45.427991 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.428140 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.428352 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":489,"threads_started":1}
I20260927 05:12:45.429417 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.415196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60471 (local address 127.0.97.62:39511)
0927 05:12:45.415472 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.415477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.416203 (+ 726us) server_negotiation.cc:407] Connection header received
0927 05:12:45.416418 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.416425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.416499 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.416591 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:45.418025 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.418961 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.420589 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.420981 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.421120 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.422611 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:45.422617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:45.422621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:45.422657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:45.425897 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.426815 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.426823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.426829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.427006 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.427444 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.427447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.427449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.427695 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:45.427854 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.429011 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.429151 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 05:12:45.440459 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44665
I20260927 05:12:45.440486 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44665 every 8 connection(s)
I20260927 05:12:45.441623 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:12:45.442828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 1861
I20260927 05:12:45.442984 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:12:45.443542 388 external_mini_cluster.cc:1428] Setting key 61f408fed526d234537450b354afcf7a
I20260927 05:12:45.443574 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39511
I20260927 05:12:45.443773 1996 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:45.444521 1996 heartbeater.cc:511] Master 127.0.97.62:39511 requested a full tablet report, sending...
I20260927 05:12:45.446003 1511 ts_manager.cc:194] Registered new tserver with Master: 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3:44665)
I20260927 05:12:45.446834 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45879
I20260927 05:12:45.458738 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:12:45.461613 1718 heartbeater.cc:503] Master 127.0.97.62:39511 was elected leader, sending a full tablet report...
Sep 27 05:12:45 dist-test-slave-1t6p krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:12:45 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:45.501710 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.488907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.97.62:39511)
0927 05:12:45.489130 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.489136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.489323 (+ 187us) server_negotiation.cc:407] Connection header received
0927 05:12:45.489530 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.489543 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.489628 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.489736 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:45.491090 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.491913 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.493149 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.493564 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.493689 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.497420 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:45.497425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:45.497429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:45.497458 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:45.499758 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.500284 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.500288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.500290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.500393 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.500694 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.500697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.500699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.500929 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:45.501099 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.501321 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.501472 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:12:45.505506 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:45.512120 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:45.534578 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.523314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38849 (local address 127.0.0.1:55320)
0927 05:12:45.524306 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.524342 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.524490 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.525682 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.525690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.525720 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.526109 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.526127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.528182 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.528190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.531254 (+ 3064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.531273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.533607 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.533618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.533851 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.533917 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.534042 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.534227 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":758,"threads_started":1}
I20260927 05:12:45.535512 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.523907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.97.1:38849)
0927 05:12:45.525177 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.525187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.525222 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:12:45.525304 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.525310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.525477 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.525750 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.526373 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.527995 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.531523 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.533418 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.534793 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.534985 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.535075 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.535250 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":980,"threads_started":1}
I20260927 05:12:45.539132 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2e5696764cb7bc681e95e9add689 (DEFAULT_TABLE table=test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.540437 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2e5696764cb7bc681e95e9add689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.540799 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.524019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.97.2:41385)
0927 05:12:45.526891 (+ 2872us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.526900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.526925 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:12:45.527036 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.527043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.527254 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.533483 (+ 6229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.534304 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.535698 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.538207 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.539538 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.539684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.539874 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.540291 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.540444 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2722,"thread_start_us":3416,"threads_started":1}
I20260927 05:12:45.541385 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.523551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41385 (local address 127.0.0.1:57536)
0927 05:12:45.526347 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.526363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.526462 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.533473 (+ 7011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.533478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.533507 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.533935 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.533946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.535898 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.535903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.538040 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.538055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.539879 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.539886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.540044 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.540083 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.540879 (+ 796us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.540997 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2664,"mutex_wait_us":57,"thread_start_us":3465,"threads_started":1}
I20260927 05:12:45.543202 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.527366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44665 (local address 127.0.0.1:45042)
0927 05:12:45.528046 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.528062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.528164 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.535300 (+ 7136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.535307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.535336 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.535888 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.535902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.537718 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.537724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.539738 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.539754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.542380 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.542386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.542540 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.542570 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.542667 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.542798 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":3,"thread_start_us":450,"threads_started":1}
I20260927 05:12:45.543243 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.527595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.97.3:44665)
0927 05:12:45.534066 (+ 6471us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.534074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.534101 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:12:45.534182 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.534189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.534395 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.534559 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.536152 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.537581 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.539918 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.541330 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.541468 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.541696 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.542726 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.542890 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6292,"thread_start_us":6469,"threads_started":1}
I20260927 05:12:45.544575 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2e5696764cb7bc681e95e9add689 (DEFAULT_TABLE table=test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.545918 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2e5696764cb7bc681e95e9add689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.546765 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2e5696764cb7bc681e95e9add689 (DEFAULT_TABLE table=test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.548096 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2e5696764cb7bc681e95e9add689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.552127 2015 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Bootstrap starting.
I20260927 05:12:45.555951 2015 tablet_bootstrap.cc:654] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.556932 2015 log.cc:824] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:45.559437 2015 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: No bootstrap required, opened a new log
I20260927 05:12:45.559692 2015 ts_tablet_manager.cc:1397] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:12:45.559975 2017 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Bootstrap starting.
I20260927 05:12:45.562624 2018 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Bootstrap starting.
I20260927 05:12:45.563253 2017 tablet_bootstrap.cc:654] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.564668 2017 log.cc:824] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:45.566021 2018 tablet_bootstrap.cc:654] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.566900 2018 log.cc:824] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:45.567591 2017 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: No bootstrap required, opened a new log
I20260927 05:12:45.567835 2017 ts_tablet_manager.cc:1397] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 05:12:45.569495 2018 tablet_bootstrap.cc:492] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: No bootstrap required, opened a new log
I20260927 05:12:45.569737 2018 ts_tablet_manager.cc:1397] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 05:12:45.571302 2015 raft_consensus.cc:348] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.571682 2015 raft_consensus.cc:374] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.571782 2015 raft_consensus.cc:729] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad9286535934642852fae90a56c9683, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.572767 2015 consensus_queue.cc:260] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.574324 2015 ts_tablet_manager.cc:1428] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 05:12:45.577922 2017 raft_consensus.cc:348] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.578316 2017 raft_consensus.cc:374] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.578416 2017 raft_consensus.cc:729] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f3478ba0b84446919470f9b95c09a9, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.578369 2018 raft_consensus.cc:348] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.578763 2018 raft_consensus.cc:374] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.578866 2018 raft_consensus.cc:729] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f7ce2e5d347d0a28f752718462b71, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.579178 2017 consensus_queue.cc:260] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.579456 2018 consensus_queue.cc:260] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.580476 1857 heartbeater.cc:503] Master 127.0.97.62:39511 was elected leader, sending a full tablet report...
I20260927 05:12:45.580881 2017 ts_tablet_manager.cc:1428] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260927 05:12:45.580900 1996 heartbeater.cc:503] Master 127.0.97.62:39511 was elected leader, sending a full tablet report...
I20260927 05:12:45.581447 2018 ts_tablet_manager.cc:1428] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 05:12:45.623833 2021 raft_consensus.cc:482] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:45.624121 2021 raft_consensus.cc:504] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.625794 2021 leader_election.cc:288] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385), 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3:44665)
I20260927 05:12:45.632902 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.626226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44075 (local address 127.0.97.2:41385)
0927 05:12:45.626440 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.626446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.626668 (+ 222us) server_negotiation.cc:407] Connection header received
0927 05:12:45.627048 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.627054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.627127 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.627190 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.628379 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.629143 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.630959 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.631935 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.632087 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.632189 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.632513 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.632631 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:12:45.633036 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.625924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41385 (local address 127.0.97.1:44075)
0927 05:12:45.626570 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.626589 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.626826 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.627396 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.627404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.627430 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.627834 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.627846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.629277 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.629292 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.630821 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.630832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.632133 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.632139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.632324 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.632347 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.632488 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.632631 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":43,"thread_start_us":294,"threads_started":1}
I20260927 05:12:45.633316 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.626597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44665 (local address 127.0.97.1:46585)
0927 05:12:45.627234 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.627252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.627366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.627636 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.627640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.627661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.628506 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.628515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.629596 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.629601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.630978 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.630990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.632386 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.632393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.632610 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.632642 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.632750 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.632866 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":4,"thread_start_us":276,"threads_started":1}
I20260927 05:12:45.633440 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.626909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46585 (local address 127.0.97.3:44665)
0927 05:12:45.627112 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.627116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.627321 (+ 205us) server_negotiation.cc:407] Connection header received
0927 05:12:45.627461 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.627465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.627517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.627590 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.628652 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.629447 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.631181 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.632169 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.632308 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.632416 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.632780 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.632896 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:12:45.634166 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2e5696764cb7bc681e95e9add689" candidate_uuid: "8ad9286535934642852fae90a56c9683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9" is_pre_election: true
I20260927 05:12:45.634270 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2e5696764cb7bc681e95e9add689" candidate_uuid: "8ad9286535934642852fae90a56c9683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386f7ce2e5d347d0a28f752718462b71" is_pre_election: true
I20260927 05:12:45.634650 1807 raft_consensus.cc:2454] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad9286535934642852fae90a56c9683 in term 0.
I20260927 05:12:45.634764 1951 raft_consensus.cc:2454] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad9286535934642852fae90a56c9683 in term 0.
I20260927 05:12:45.635306 1600 leader_election.cc:302] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad9286535934642852fae90a56c9683, f8f3478ba0b84446919470f9b95c09a9; no voters:
I20260927 05:12:45.635685 2021 raft_consensus.cc:2781] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:45.635831 2021 raft_consensus.cc:482] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:45.635953 2021 raft_consensus.cc:3037] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.638089 2021 raft_consensus.cc:504] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.638618 2021 leader_election.cc:288] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [CANDIDATE]: Term 1 election: Requested vote from peers f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385), 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3:44665)
I20260927 05:12:45.639034 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2e5696764cb7bc681e95e9add689" candidate_uuid: "8ad9286535934642852fae90a56c9683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9"
I20260927 05:12:45.639096 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2e5696764cb7bc681e95e9add689" candidate_uuid: "8ad9286535934642852fae90a56c9683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386f7ce2e5d347d0a28f752718462b71"
I20260927 05:12:45.639225 1807 raft_consensus.cc:3037] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.639294 1951 raft_consensus.cc:3037] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.641666 1807 raft_consensus.cc:2454] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad9286535934642852fae90a56c9683 in term 1.
I20260927 05:12:45.641803 1951 raft_consensus.cc:2454] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad9286535934642852fae90a56c9683 in term 1.
I20260927 05:12:45.642032 1600 leader_election.cc:302] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad9286535934642852fae90a56c9683, f8f3478ba0b84446919470f9b95c09a9; no voters:
I20260927 05:12:45.642306 2021 raft_consensus.cc:2781] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:45.642695 2021 raft_consensus.cc:686] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad9286535934642852fae90a56c9683, State: Running, Role: LEADER
I20260927 05:12:45.643265 2021 consensus_queue.cc:237] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.647480 1511 catalog_manager.cc:5495] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad9286535934642852fae90a56c9683 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8ad9286535934642852fae90a56c9683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } health_report { overall_health: UNKNOWN } } }
I20260927 05:12:45.691339 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.685862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.97.1:38849)
0927 05:12:45.686044 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.686052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.686230 (+ 178us) server_negotiation.cc:407] Connection header received
0927 05:12:45.686528 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.686532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.686613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.686729 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:45.687040 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.687776 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.689011 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.689787 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.689877 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.690290 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:45.690753 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:45.690857 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.690939 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.691037 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 05:12:45.693171 1997 tablet.cc:2369] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:12:45.697217 1858 tablet.cc:2369] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:12:45.699358 1807 raft_consensus.cc:1261] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Refusing update from remote peer 8ad9286535934642852fae90a56c9683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:45.699438 1951 raft_consensus.cc:1261] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Refusing update from remote peer 8ad9286535934642852fae90a56c9683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:45.700351 2026 consensus_queue.cc:1035] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:45.700735 2021 consensus_queue.cc:1035] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 05:12:45.703851 1719 tablet.cc:2369] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:12:45.711997 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333830515494400000 to 7333830515278929920. Current Snapshot: MvccSnapshot[applied={T|T < 7333830515494400000}]
I20260927 05:12:45.712819 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333830515494400000 to 7333830515278929920. Current Snapshot: MvccSnapshot[applied={T|T < 7333830515494400000}]
I20260927 05:12:45.730393 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:45.730648 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:45.734176 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Sending DeleteTablet for 3 replicas of tablet 37ac2e5696764cb7bc681e95e9add689
I20260927 05:12:45.735337 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ac2e5696764cb7bc681e95e9add689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260927 05:12:45.735450 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ac2e5696764cb7bc681e95e9add689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260927 05:12:45.736181 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ac2e5696764cb7bc681e95e9add689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45042
I20260927 05:12:45.737015 2041 tablet_replica.cc:330] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: stopping tablet replica
I20260927 05:12:45.737493 2041 raft_consensus.cc:2229] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:45.738031 2041 raft_consensus.cc:2258] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:45.737882 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:45.739480 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:45.739888 2040 tablet_replica.cc:330] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: stopping tablet replica
I20260927 05:12:45.740168 2039 tablet_replica.cc:330] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: stopping tablet replica
I20260927 05:12:45.740656 2039 raft_consensus.cc:2229] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:45.740731 2041 ts_tablet_manager.cc:1905] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:45.741134 2040 raft_consensus.cc:2229] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:45.741624 2040 raft_consensus.cc:2258] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:45.742300 2039 raft_consensus.cc:2258] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:45.746228 2040 ts_tablet_manager.cc:1905] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:45.747125 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 3180c7f07e95476296af55418e807be4 (DEFAULT_TABLE table=test-table [id=d9f57fa0db964d10a31d228a793cf492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.747186 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 3180c7f07e95476296af55418e807be4 (DEFAULT_TABLE table=test-table [id=d9f57fa0db964d10a31d228a793cf492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.747565 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3180c7f07e95476296af55418e807be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.747588 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3180c7f07e95476296af55418e807be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.753026 2015 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Bootstrap starting.
I20260927 05:12:45.754019 2018 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Bootstrap starting.
I20260927 05:12:45.754568 2015 tablet_bootstrap.cc:654] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.754555 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 3180c7f07e95476296af55418e807be4 (DEFAULT_TABLE table=test-table [id=d9f57fa0db964d10a31d228a793cf492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:45.754992 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3180c7f07e95476296af55418e807be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.755578 2018 tablet_bootstrap.cc:654] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.760998 2040 ts_tablet_manager.cc:1918] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:12:45.761221 2040 log.cc:1196] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/37ac2e5696764cb7bc681e95e9add689
I20260927 05:12:45.761909 2040 ts_tablet_manager.cc:1939] T 37ac2e5696764cb7bc681e95e9add689 P f8f3478ba0b84446919470f9b95c09a9: Deleting consensus metadata
I20260927 05:12:45.762975 2017 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Bootstrap starting.
I20260927 05:12:45.764197 1497 catalog_manager.cc:4841] TS f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385): tablet 37ac2e5696764cb7bc681e95e9add689 (table test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]) successfully deleted
I20260927 05:12:45.764578 2017 tablet_bootstrap.cc:654] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.764837 2039 ts_tablet_manager.cc:1905] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:45.775942 2041 ts_tablet_manager.cc:1918] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:12:45.776115 2041 log.cc:1196] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/37ac2e5696764cb7bc681e95e9add689
I20260927 05:12:45.777702 2015 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: No bootstrap required, opened a new log
I20260927 05:12:45.777773 2041 ts_tablet_manager.cc:1939] T 37ac2e5696764cb7bc681e95e9add689 P 386f7ce2e5d347d0a28f752718462b71: Deleting consensus metadata
I20260927 05:12:45.777876 2015 ts_tablet_manager.cc:1397] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260927 05:12:45.778918 2017 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: No bootstrap required, opened a new log
I20260927 05:12:45.779090 2017 ts_tablet_manager.cc:1397] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260927 05:12:45.779146 2015 raft_consensus.cc:348] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.779434 2015 raft_consensus.cc:374] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.779529 2015 raft_consensus.cc:729] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad9286535934642852fae90a56c9683, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.779584 1497 catalog_manager.cc:4841] TS 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3:44665): tablet 37ac2e5696764cb7bc681e95e9add689 (table test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]) successfully deleted
I20260927 05:12:45.779830 2015 consensus_queue.cc:260] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.779881 2017 raft_consensus.cc:348] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.780114 2017 raft_consensus.cc:374] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.780210 2017 raft_consensus.cc:729] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f3478ba0b84446919470f9b95c09a9, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.780362 2015 ts_tablet_manager.cc:1428] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:45.780493 2017 consensus_queue.cc:260] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.780630 2039 ts_tablet_manager.cc:1918] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:12:45.780778 2039 log.cc:1196] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/37ac2e5696764cb7bc681e95e9add689
I20260927 05:12:45.781316 2039 ts_tablet_manager.cc:1939] T 37ac2e5696764cb7bc681e95e9add689 P 8ad9286535934642852fae90a56c9683: Deleting consensus metadata
I20260927 05:12:45.781816 2018 tablet_bootstrap.cc:492] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: No bootstrap required, opened a new log
I20260927 05:12:45.781996 2018 ts_tablet_manager.cc:1397] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.012s
I20260927 05:12:45.782869 1499 catalog_manager.cc:4841] TS 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849): tablet 37ac2e5696764cb7bc681e95e9add689 (table test-table [id=5711b29b2c6d4e59aa69a3d4f25b7a60]) successfully deleted
I20260927 05:12:45.782787 2018 raft_consensus.cc:348] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.783113 2018 raft_consensus.cc:374] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.783226 2018 raft_consensus.cc:729] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f7ce2e5d347d0a28f752718462b71, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.783519 2018 consensus_queue.cc:260] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.785902 2017 ts_tablet_manager.cc:1428] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 05:12:45.785967 2018 ts_tablet_manager.cc:1428] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 05:12:45.809868 2023 raft_consensus.cc:482] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:45.810096 2023 raft_consensus.cc:504] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.811740 2023 leader_election.cc:288] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849), f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385)
I20260927 05:12:45.819068 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.811911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38849 (local address 127.0.97.3:46255)
0927 05:12:45.812282 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.812303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.812422 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.813245 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.813250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.813275 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.813498 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.813510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.814993 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.815018 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.817147 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.817160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.818300 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.818306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.818484 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.818507 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.818604 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.818743 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":71}
I20260927 05:12:45.819093 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.812752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46255 (local address 127.0.97.1:38849)
0927 05:12:45.812927 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.812934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.812951 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:12:45.813020 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.813026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.813107 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.813211 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.813664 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.814843 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.817337 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.818106 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.818282 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.818382 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.818638 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.818760 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:12:45.819734 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.812131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60251 (local address 127.0.97.2:41385)
0927 05:12:45.812347 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.812356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.812961 (+ 605us) server_negotiation.cc:407] Connection header received
0927 05:12:45.813122 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.813127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.813215 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.813306 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:45.813916 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.814634 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.817303 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.818549 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.818673 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.818765 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.819343 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.819474 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:12:45.820127 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3180c7f07e95476296af55418e807be4" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad9286535934642852fae90a56c9683" is_pre_election: true
I20260927 05:12:45.820393 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.812003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41385 (local address 127.0.97.3:60251)
0927 05:12:45.812834 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.812859 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.812980 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.813375 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.813380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.813406 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:45.813780 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.813788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.814764 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.814770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.817149 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.817160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.818912 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.818920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.819159 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.819196 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.819896 (+ 700us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.820028 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":138,"thread_start_us":563,"threads_started":1}
I20260927 05:12:45.820521 1669 raft_consensus.cc:2454] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 0.
I20260927 05:12:45.820993 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3180c7f07e95476296af55418e807be4" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9" is_pre_election: true
I20260927 05:12:45.821165 1880 leader_election.cc:302] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386f7ce2e5d347d0a28f752718462b71, 8ad9286535934642852fae90a56c9683; no voters:
I20260927 05:12:45.821249 1807 raft_consensus.cc:2454] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 0.
I20260927 05:12:45.821534 2023 raft_consensus.cc:2781] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:45.821692 2023 raft_consensus.cc:482] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:45.821795 2023 raft_consensus.cc:3037] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.823889 2023 raft_consensus.cc:504] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.824544 2023 leader_election.cc:288] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849), f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385)
I20260927 05:12:45.824863 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3180c7f07e95476296af55418e807be4" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad9286535934642852fae90a56c9683"
I20260927 05:12:45.824908 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3180c7f07e95476296af55418e807be4" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9"
I20260927 05:12:45.825078 1807 raft_consensus.cc:3037] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.825076 1669 raft_consensus.cc:3037] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:45.828069 1807 raft_consensus.cc:2454] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 1.
I20260927 05:12:45.828486 1878 leader_election.cc:302] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386f7ce2e5d347d0a28f752718462b71, f8f3478ba0b84446919470f9b95c09a9; no voters:
I20260927 05:12:45.828637 1669 raft_consensus.cc:2454] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 1.
I20260927 05:12:45.828797 2023 raft_consensus.cc:2781] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:45.829428 2023 raft_consensus.cc:686] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 1 LEADER]: Becoming Leader. State: Replica: 386f7ce2e5d347d0a28f752718462b71, State: Running, Role: LEADER
I20260927 05:12:45.829778 2023 consensus_queue.cc:237] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.833292 1510 catalog_manager.cc:5495] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } health_report { overall_health: HEALTHY } } }
Sep 27 05:12:45 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:45.896571 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.881360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.97.62:39511)
0927 05:12:45.881560 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:45.881567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:45.882145 (+ 578us) server_negotiation.cc:407] Connection header received
0927 05:12:45.882284 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:45.882290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:45.882366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:45.882482 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:45.883599 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.884758 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.886242 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.886933 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.887079 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.890609 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:45.890620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:45.890623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:45.890648 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:45.893460 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.894331 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.894336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.894338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.894478 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:45.894923 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:45.894927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:45.894930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:45.895277 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:45.895399 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:45.895984 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 05:12:45.896182 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:12:45.897697 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:45.881363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39511 (local address 127.0.0.1:45512)
0927 05:12:45.882029 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:45.882050 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:45.882170 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:45.882512 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:45.882515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:45.883234 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:45.883463 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.883471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.884902 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.884906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.886113 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:45.886124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:45.887175 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:45.887181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:45.887314 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:45.887973 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:45.887990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:45.890443 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:45.893603 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.893614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.893622 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.894218 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.894608 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:45.894612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:45.894615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:45.894799 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:45.895494 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:45.895503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:45.895681 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 05:12:45.897065 (+ 1384us) client_negotiation.cc:240] Negotiation successful
0927 05:12:45.897359 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":385,"threads_started":1}
I20260927 05:12:45.903159 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45512:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:12:45.904261 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:45.910563 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c60454401471d80d0b37726d29b8d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfb1502c5e854959836de4d4527482c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:45.911144 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c60454401471d80d0b37726d29b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.915867 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c60454401471d80d0b37726d29b8d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfb1502c5e854959836de4d4527482c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:45.915954 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c60454401471d80d0b37726d29b8d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfb1502c5e854959836de4d4527482c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:45.916469 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c60454401471d80d0b37726d29b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.916777 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c60454401471d80d0b37726d29b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:45.917906 2015 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: Bootstrap starting.
I20260927 05:12:45.919492 2015 tablet_bootstrap.cc:654] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.923257 2018 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: Bootstrap starting.
I20260927 05:12:45.924932 2018 tablet_bootstrap.cc:654] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.925123 2017 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: Bootstrap starting.
I20260927 05:12:45.927256 2017 tablet_bootstrap.cc:654] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:45.934041 2015 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: No bootstrap required, opened a new log
I20260927 05:12:45.934201 2015 ts_tablet_manager.cc:1397] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260927 05:12:45.934885 2015 raft_consensus.cc:348] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.935190 2015 raft_consensus.cc:374] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.935293 2015 raft_consensus.cc:729] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad9286535934642852fae90a56c9683, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.935647 2015 consensus_queue.cc:260] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.936429 2015 ts_tablet_manager.cc:1428] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:45.936179 2034 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:45.936585 2034 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:45.938086 2018 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: No bootstrap required, opened a new log
I20260927 05:12:45.938259 2018 ts_tablet_manager.cc:1397] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260927 05:12:45.938356 2017 tablet_bootstrap.cc:492] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: No bootstrap required, opened a new log
I20260927 05:12:45.938509 2017 ts_tablet_manager.cc:1397] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260927 05:12:45.939033 2018 raft_consensus.cc:348] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.939278 2018 raft_consensus.cc:374] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.939194 2017 raft_consensus.cc:348] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.939357 2018 raft_consensus.cc:729] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f7ce2e5d347d0a28f752718462b71, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.939426 2017 raft_consensus.cc:374] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:45.939495 2017 raft_consensus.cc:729] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f3478ba0b84446919470f9b95c09a9, State: Initialized, Role: FOLLOWER
I20260927 05:12:45.939628 2018 consensus_queue.cc:260] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.939761 2017 consensus_queue.cc:260] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:45.940061 2023 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:45.940752 2023 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:45.940743 2017 ts_tablet_manager.cc:1428] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:45.940217 2022 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:45.940956 2022 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:45.941167 2018 ts_tablet_manager.cc:1428] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 05:12:46.092829 2023 raft_consensus.cc:482] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:46.093067 2023 raft_consensus.cc:504] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:46.093706 2023 leader_election.cc:288] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849), f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385)
I20260927 05:12:46.094270 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c60454401471d80d0b37726d29b8d" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9" is_pre_election: true
I20260927 05:12:46.094280 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c60454401471d80d0b37726d29b8d" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad9286535934642852fae90a56c9683" is_pre_election: true
I20260927 05:12:46.094530 1807 raft_consensus.cc:2454] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 0.
I20260927 05:12:46.094586 1669 raft_consensus.cc:2454] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 0.
I20260927 05:12:46.094947 1878 leader_election.cc:302] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386f7ce2e5d347d0a28f752718462b71, f8f3478ba0b84446919470f9b95c09a9; no voters:
I20260927 05:12:46.095249 2023 raft_consensus.cc:2781] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:46.095357 2023 raft_consensus.cc:482] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:46.095466 2023 raft_consensus.cc:3037] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:46.097493 2023 raft_consensus.cc:504] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:46.098047 2023 leader_election.cc:288] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849), f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385)
I20260927 05:12:46.098492 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c60454401471d80d0b37726d29b8d" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad9286535934642852fae90a56c9683"
I20260927 05:12:46.098558 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c60454401471d80d0b37726d29b8d" candidate_uuid: "386f7ce2e5d347d0a28f752718462b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f3478ba0b84446919470f9b95c09a9"
I20260927 05:12:46.098703 1669 raft_consensus.cc:3037] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:46.098793 1807 raft_consensus.cc:3037] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:46.100878 1669 raft_consensus.cc:2454] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 1.
I20260927 05:12:46.101257 1880 leader_election.cc:302] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386f7ce2e5d347d0a28f752718462b71, 8ad9286535934642852fae90a56c9683; no voters:
I20260927 05:12:46.101518 2023 raft_consensus.cc:2781] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:46.101788 1807 raft_consensus.cc:2454] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386f7ce2e5d347d0a28f752718462b71 in term 1.
I20260927 05:12:46.102116 2023 raft_consensus.cc:686] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [term 1 LEADER]: Becoming Leader. State: Replica: 386f7ce2e5d347d0a28f752718462b71, State: Running, Role: LEADER
I20260927 05:12:46.102370 2023 consensus_queue.cc:237] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } }
I20260927 05:12:46.104040 2046 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: TxnStatusTablet state changed. Reason: New leader 386f7ce2e5d347d0a28f752718462b71. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.104314 2046 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:46.104903 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:12:46.105113 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:12:46.105638 1511 catalog_manager.cc:5495] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } health_report { overall_health: HEALTHY } } }
I20260927 05:12:46.167716 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.159787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44665 (local address 127.0.0.1:45054)
0927 05:12:46.160002 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:46.160025 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:46.160185 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:46.160943 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:46.160949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:46.160965 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:46.161160 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:46.161169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.163271 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.163279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:46.164893 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:46.164908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.165852 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.165858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:46.166034 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.166104 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:46.167175 (+ 1071us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:46.167197 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:12:46.167274 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 05:12:46.167430 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 05:12:46.167747 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.159950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.97.3:44665)
0927 05:12:46.160509 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.160515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.160535 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:12:46.160628 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.160633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.160735 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.160898 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:46.161343 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.163095 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.165044 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.165686 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.165811 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.166296 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:46.167051 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:46.167242 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.167312 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.167388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260927 05:12:46.172261 1807 raft_consensus.cc:1261] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Refusing update from remote peer 386f7ce2e5d347d0a28f752718462b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:46.172524 1669 raft_consensus.cc:1261] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Refusing update from remote peer 386f7ce2e5d347d0a28f752718462b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:46.172984 2023 consensus_queue.cc:1035] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:46.175144 2023 consensus_queue.cc:1035] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:46.186197 2034 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: TxnStatusTablet state changed. Reason: New leader 386f7ce2e5d347d0a28f752718462b71. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.186499 2022 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: TxnStatusTablet state changed. Reason: New leader 386f7ce2e5d347d0a28f752718462b71. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.186906 2022 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:46.186923 2034 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:46.188668 2023 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.188948 2023 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:46.192445 2034 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.192658 2034 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 8ad9286535934642852fae90a56c9683: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:46.197227 2046 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.197952 2046 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P 386f7ce2e5d347d0a28f752718462b71: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:46.201669 2022 tablet_replica.cc:439] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "386f7ce2e5d347d0a28f752718462b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 } } peers { permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 } } peers { permanent_uuid: "386f7ce2e5d347d0a28f752718462b71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44665 } } }
I20260927 05:12:46.201897 2022 tablet_replica.cc:442] T 1d7c60454401471d80d0b37726d29b8d P f8f3478ba0b84446919470f9b95c09a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:46.216952 2046 consensus_queue.cc:1035] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad9286535934642852fae90a56c9683" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:46.229074 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.196631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.97.3:44665)
0927 05:12:46.196817 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.196824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.214893 (+ 18069us) server_negotiation.cc:407] Connection header received
0927 05:12:46.215183 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.215188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.215270 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.215327 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:46.223858 (+ 8531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.224963 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.226949 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.227597 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.227694 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.228199 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:46.228345 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:46.228458 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.228624 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.228733 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:12:46.232565 2046 consensus_queue.cc:1035] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f3478ba0b84446919470f9b95c09a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:46.252732 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.246341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44289 (local address 127.0.97.3:44665)
0927 05:12:46.246504 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.246508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.246946 (+ 438us) server_negotiation.cc:407] Connection header received
0927 05:12:46.247162 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.247167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.247262 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.247355 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:46.248273 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.249421 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.250786 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.251465 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.251585 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.251970 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:46.252090 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:46.252182 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.252306 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.252378 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 05:12:46.252755 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.246248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44665 (local address 127.0.97.3:44289)
0927 05:12:46.246848 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:46.246867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:46.246978 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:46.247378 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:46.247380 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:46.247390 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:46.248098 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:46.248107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.249542 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.249545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:46.250657 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:46.250667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.251633 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.251637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:46.251753 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.251812 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:46.252188 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:46.252195 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:12:46.252252 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 05:12:46.252361 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
I20260927 05:12:46.320545 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:46.320813 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:46.323797 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e3e4bde0ddd41f78701d78e9dbf5ab6: Sending DeleteTablet for 3 replicas of tablet 3180c7f07e95476296af55418e807be4
I20260927 05:12:46.324457 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 3180c7f07e95476296af55418e807be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260927 05:12:46.324985 2074 tablet_replica.cc:330] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: stopping tablet replica
I20260927 05:12:46.324931 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 3180c7f07e95476296af55418e807be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260927 05:12:46.325174 2074 raft_consensus.cc:2229] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:46.325426 2074 raft_consensus.cc:2258] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:46.325482 2075 tablet_replica.cc:330] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: stopping tablet replica
I20260927 05:12:46.325778 2075 raft_consensus.cc:2229] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:46.325970 2075 raft_consensus.cc:2258] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:46.326855 2075 ts_tablet_manager.cc:1905] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:46.327029 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 3180c7f07e95476296af55418e807be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45042
I20260927 05:12:46.327713 2077 tablet_replica.cc:330] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: stopping tablet replica
I20260927 05:12:46.329365 2074 ts_tablet_manager.cc:1905] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:46.334228 2077 raft_consensus.cc:2229] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:46.334556 2075 ts_tablet_manager.cc:1918] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:46.334682 2075 log.cc:1196] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/3180c7f07e95476296af55418e807be4
I20260927 05:12:46.335155 2075 ts_tablet_manager.cc:1939] T 3180c7f07e95476296af55418e807be4 P f8f3478ba0b84446919470f9b95c09a9: Deleting consensus metadata
I20260927 05:12:46.335536 2077 raft_consensus.cc:2258] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:46.336136 1497 catalog_manager.cc:4841] TS f8f3478ba0b84446919470f9b95c09a9 (127.0.97.2:41385): tablet 3180c7f07e95476296af55418e807be4 (table test-table [id=d9f57fa0db964d10a31d228a793cf492]) successfully deleted
I20260927 05:12:46.337075 2077 ts_tablet_manager.cc:1905] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:46.339087 2074 ts_tablet_manager.cc:1918] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:46.339233 2074 log.cc:1196] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/3180c7f07e95476296af55418e807be4
I20260927 05:12:46.339653 2074 ts_tablet_manager.cc:1939] T 3180c7f07e95476296af55418e807be4 P 8ad9286535934642852fae90a56c9683: Deleting consensus metadata
I20260927 05:12:46.344128 1499 catalog_manager.cc:4841] TS 8ad9286535934642852fae90a56c9683 (127.0.97.1:38849): tablet 3180c7f07e95476296af55418e807be4 (table test-table [id=d9f57fa0db964d10a31d228a793cf492]) successfully deleted
I20260927 05:12:46.348217 2077 ts_tablet_manager.cc:1918] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:46.348409 2077 log.cc:1196] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/3180c7f07e95476296af55418e807be4
I20260927 05:12:46.348903 2077 ts_tablet_manager.cc:1939] T 3180c7f07e95476296af55418e807be4 P 386f7ce2e5d347d0a28f752718462b71: Deleting consensus metadata
I20260927 05:12:46.349941 1497 catalog_manager.cc:4841] TS 386f7ce2e5d347d0a28f752718462b71 (127.0.97.3:44665): tablet 3180c7f07e95476296af55418e807be4 (table test-table [id=d9f57fa0db964d10a31d228a793cf492]) successfully deleted
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:46.361101 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.328138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.97.1:38849)
0927 05:12:46.344118 (+ 15980us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.344125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.344142 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:12:46.344194 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.344199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.344288 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.344391 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:46.346812 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.348001 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.349823 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.350463 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.350587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.354886 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:46.354898 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:46.354912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:46.354943 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:46.358499 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.359230 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.359238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.359240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.359378 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.359747 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.359749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.359751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.360094 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:46.360302 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.360621 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.360782 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15852,"thread_start_us":15747,"threads_started":1}
W20260927 05:12:46.361822 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55332
I20260927 05:12:46.378263 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.364761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.0.97.62:39511)
0927 05:12:46.364946 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.364952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.365196 (+ 244us) server_negotiation.cc:407] Connection header received
0927 05:12:46.365295 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.365300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.365392 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.365497 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:46.366627 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.367803 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.369151 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.370298 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.370421 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.372486 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:46.372494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:46.372499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:46.372541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:46.375737 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.376330 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.376336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.376340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.376512 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.376864 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.376869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.376873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.377195 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:46.377441 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.377711 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.377925 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 05:12:46.378787 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45528
I20260927 05:12:46.381807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1583
I20260927 05:12:46.395290 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.382515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.97.1:38849)
0927 05:12:46.382778 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.382787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.383258 (+ 471us) server_negotiation.cc:407] Connection header received
0927 05:12:46.383381 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.383386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.383455 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.383541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:46.384646 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.385621 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.386791 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.387437 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.387577 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.389704 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:46.389711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:46.389717 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:46.389756 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:46.392796 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.393465 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.393470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.393474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.393612 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.393929 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.393933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.393937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.394264 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:46.394503 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.394677 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.394885 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 05:12:46.395900 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55340
W20260927 05:12:46.396417 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:46.396638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1583
I20260927 05:12:46.425694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1722
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:12:46.440891 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.426447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.97.2:41385)
0927 05:12:46.426933 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.426938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.426952 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:12:46.427047 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.427052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.427132 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.427243 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:46.428432 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.429281 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.430452 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.430851 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.430976 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.435074 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:46.435091 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:46.435110 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:46.435142 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:46.438074 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.438703 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.438715 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.438718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.438894 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.439260 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.439265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.439269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.439591 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:46.439896 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.440283 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.440501 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
W20260927 05:12:46.443708 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57544
W20260927 05:12:46.444336 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:46.444489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1722
I20260927 05:12:46.461468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1861
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:12:46.477264 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:46.462176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.97.3:44665)
0927 05:12:46.462363 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:46.462369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:46.462519 (+ 150us) server_negotiation.cc:407] Connection header received
0927 05:12:46.462712 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:46.462717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:46.462805 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:46.462916 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:46.464131 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.464898 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.466190 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:46.466685 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:46.466833 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:46.471170 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:46.471180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:46.471195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:46.471225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:46.474269 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.474945 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.474954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.474959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.475286 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:46.475659 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:46.475667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:46.475670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:46.476021 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:46.476375 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:46.476633 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 05:12:46.476838 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 05:12:46.477963 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072
W20260927 05:12:46.478678 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:46.478855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1861
I20260927 05:12:46.498461 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 1480
I20260927 05:12:46.499550 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:46.610532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 1480
2026-09-27T05:12:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4400 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[2087](info): set up 2 sockets
Sep 27 05:12:46 dist-test-slave-1t6p krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260927 05:12:47.687129 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:12:47 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:47.858755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:48.170782 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:48.171118 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:48.171283 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:48.178586 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:48.178750 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:12:48.178854 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:48.178972 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:48.179077 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:48.179188 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:48.189555 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39731
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:48.190835 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:48.192160 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:48.203675 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:48.203616 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:48.203651 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:48.204555 2100 server_base.cc:1034] running on GCE node
I20260927 05:12:48.205240 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:48.206539 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:48.207778 2100 hybrid_clock.cc:648] HybridClock initialized: now 1790485968207714 us; error 32 us; skew 500 ppm
I20260927 05:12:48.208086 2100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:48 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:48.220522 2100 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:48.221987 2100 webserver.cc:460] Webserver started at http://127.0.97.62:33495/ using document root <none> and password file <none>
I20260927 05:12:48.222663 2100 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:48.222818 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:48.223194 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:48.226223 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "4643aeae99494dcc8c80f1ec069bc012"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "8fa86e41512ff7b9061d3f9ea71db5c3"
server_key_iv: "cff408040fac372e8ac72e8b28d3c49b"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.227166 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "4643aeae99494dcc8c80f1ec069bc012"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "8fa86e41512ff7b9061d3f9ea71db5c3"
server_key_iv: "cff408040fac372e8ac72e8b28d3c49b"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.231905 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:12:48.234951 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:48.236232 2100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:48.236394 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "4643aeae99494dcc8c80f1ec069bc012"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "8fa86e41512ff7b9061d3f9ea71db5c3"
server_key_iv: "cff408040fac372e8ac72e8b28d3c49b"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.236605 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:48.253005 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:48.256825 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:48.257087 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:48.277514 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39731
I20260927 05:12:48.277537 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39731 every 8 connection(s)
I20260927 05:12:48.278769 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:12:48.282233 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:48.284106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2100
I20260927 05:12:48.284327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:12:48.284917 388 external_mini_cluster.cc:1428] Setting key a582446b7b05dd932c3715b48d379fe9
I20260927 05:12:48.293571 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Bootstrap starting.
I20260927 05:12:48.297122 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:48.298429 2177 log.cc:824] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:12:48 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:48.301054 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: No bootstrap required, opened a new log
I20260927 05:12:48.308476 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:48.286751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.0.97.62:39731)
0927 05:12:48.287535 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:48.287545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:48.287586 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:12:48.289187 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:48.289207 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:48.289546 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:48.289852 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:48.291557 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.292783 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.294507 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.295637 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.295770 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:48.300452 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:48.300460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:48.300473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:48.300502 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:48.304360 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.304958 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.304967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.304975 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.305167 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.305501 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.305508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.305513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.306155 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:48.306442 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:48.306745 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:12:48.307207 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":405,"threads_started":1}
I20260927 05:12:48.310940 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } }
I20260927 05:12:48.311311 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:48.311385 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4643aeae99494dcc8c80f1ec069bc012, State: Initialized, Role: FOLLOWER
I20260927 05:12:48.311821 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } }
I20260927 05:12:48.312003 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:12:48.312095 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:12:48.312207 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:48.314102 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } }
I20260927 05:12:48.314606 2177 leader_election.cc:302] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4643aeae99494dcc8c80f1ec069bc012; no voters:
I20260927 05:12:48.315263 2177 leader_election.cc:288] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:12:48.315384 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:48.316108 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [term 1 LEADER]: Becoming Leader. State: Replica: 4643aeae99494dcc8c80f1ec069bc012, State: Running, Role: LEADER
I20260927 05:12:48.316601 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } }
I20260927 05:12:48.317189 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:12:48.319774 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4643aeae99494dcc8c80f1ec069bc012. Latest consensus state: current_term: 1 leader_uuid: "4643aeae99494dcc8c80f1ec069bc012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } } }
I20260927 05:12:48.319751 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4643aeae99494dcc8c80f1ec069bc012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643aeae99494dcc8c80f1ec069bc012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39731 } } }
I20260927 05:12:48.320169 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:48.320169 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:48.321602 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:12:48.326124 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:12:48.333806 2190 catalog_manager.cc:1347] Generated new cluster ID: 91cea0b9369d453fad60c0c95a32ebe7
I20260927 05:12:48.333899 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:12:48.345477 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:12:48.346729 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:12:48.354245 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Generated new TSK 0
I20260927 05:12:48.355090 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:12:48.400656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39731
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:48.716388 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:48.716697 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:48.716892 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:48.724220 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:48.724372 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:48.724516 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:12:48.724606 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:48.724736 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:12:48.733886 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:48.735301 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:48.736555 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:48.749104 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:48.749161 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:48.749188 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:48.750689 2203 server_base.cc:1034] running on GCE node
I20260927 05:12:48.751462 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:48.752391 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:48.753618 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790485968753581 us; error 56 us; skew 500 ppm
I20260927 05:12:48.753927 2203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:48 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:48.757040 2203 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:12:48.758415 2203 webserver.cc:460] Webserver started at http://127.0.97.1:34355/ using document root <none> and password file <none>
I20260927 05:12:48.759146 2203 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:48.759289 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:48.759615 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:48.762418 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "edc9de6f53254bfea076e220b4a4edee"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "7aa11d686e840148c7d46c4f02c43d66"
server_key_iv: "1d6a17cd44dff5069e287eda4f76bc8c"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.763285 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "edc9de6f53254bfea076e220b4a4edee"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "7aa11d686e840148c7d46c4f02c43d66"
server_key_iv: "1d6a17cd44dff5069e287eda4f76bc8c"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.768251 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:12:48.771646 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:48.772930 2203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:12:48.773099 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "edc9de6f53254bfea076e220b4a4edee"
format_stamp: "Formatted at 2026-09-27 05:12:48 on dist-test-slave-1t6p"
server_key: "7aa11d686e840148c7d46c4f02c43d66"
server_key_iv: "1d6a17cd44dff5069e287eda4f76bc8c"
server_key_version: "encryptionkey@0"
I20260927 05:12:48.773283 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:48.806418 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:48.810721 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:48.811045 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:48.813413 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:48.813551 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:48.813663 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:48.813747 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:48 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:48.835062 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:48.814095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35361 (local address 127.0.97.62:39731)
0927 05:12:48.814618 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:48.814623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:48.815665 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 05:12:48.817828 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:48.817833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:48.817915 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:48.818044 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:48.820080 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.821247 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.823997 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.824410 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.824540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:48.829163 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:48.829182 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:48.829189 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:48.829242 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:48.832225 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.832904 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.832909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.832913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.833028 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.833437 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.833442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.833445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.833689 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:48.833860 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:48.834462 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 05:12:48.834657 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":315,"threads_started":1}
I20260927 05:12:48.836246 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:48.814730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.1:35361)
0927 05:12:48.815518 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:48.815552 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:48.817564 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:48.818303 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:48.818319 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:48.819057 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:48.819897 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:48.819922 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.821423 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.821427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:48.823855 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:48.823866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.824635 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.824645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:48.824959 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:48.825775 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:48.825817 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:48.828942 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:48.832353 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:48.832361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:48.832378 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:48.832783 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:48.833136 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:48.833141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:48.833145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:48.833323 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:48.833819 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:48.833825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:48.834136 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 05:12:48.834375 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 05:12:48.834834 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":364,"threads_started":1}
I20260927 05:12:48.856751 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:48.842445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44403 (local address 127.0.97.62:39731)
0927 05:12:48.842597 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:48.842601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:48.843434 (+ 833us) server_negotiation.cc:407] Connection header received
0927 05:12:48.843633 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:48.843639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:48.843720 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:48.843820 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:48.845184 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.846011 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.847695 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.848529 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.848668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:48.850391 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:48.850401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:48.850408 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:48.850455 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:48.853754 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.854775 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.854779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.854781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.854898 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:48.855357 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:48.855362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:48.855366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:48.855617 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:48.855722 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:48.856305 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 05:12:48.856504 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:12:48.857259 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:48.842580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.1:44403)
0927 05:12:48.843327 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:48.843348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:48.843503 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:48.843879 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:48.843887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:48.844595 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:48.845031 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:48.845047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.846148 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.846154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:48.847544 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:48.847564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:48.848714 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:48.848719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:48.848835 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:48.849463 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:48.849478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:48.850165 (+ 687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:48.853900 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:48.853906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:48.853910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:48.854649 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:48.855031 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:48.855039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:48.855045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:48.855229 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:48.855734 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:48.855741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:48.855996 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 05:12:48.856170 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 05:12:48.856600 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
I20260927 05:12:48.867265 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37569
I20260927 05:12:48.867306 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37569 every 8 connection(s)
I20260927 05:12:48.868462 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:12:48.870324 2339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39731
I20260927 05:12:48.870558 2339 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:48.871236 2339 heartbeater.cc:511] Master 127.0.97.62:39731 requested a full tablet report, sending...
I20260927 05:12:48.873009 2132 ts_manager.cc:194] Registered new tserver with Master: edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569)
I20260927 05:12:48.874356 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35361
I20260927 05:12:48.878252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2203
I20260927 05:12:48.878561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:12:48.879230 388 external_mini_cluster.cc:1428] Setting key 508b374244ae2b62edfe466528ee174c
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:12:48.917589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39731
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:49.223443 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:49.223704 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:49.223799 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:49.230639 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:49.230737 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:49.230818 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:12:49.230870 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:49.230935 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:12:49.240368 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:49.241592 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:49.242789 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:49.255432 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:49.255472 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:49.255502 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:49.256347 2343 server_base.cc:1034] running on GCE node
I20260927 05:12:49.256917 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:49.257779 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:49.259051 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790485969258962 us; error 52 us; skew 500 ppm
I20260927 05:12:49.259358 2343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:49.262212 2343 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:12:49.263705 2343 webserver.cc:460] Webserver started at http://127.0.97.2:41109/ using document root <none> and password file <none>
I20260927 05:12:49.264357 2343 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:49.264499 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:49.264796 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:49.267587 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "50c39a8e067c5fffad46620e403934a2"
server_key_iv: "68ab3b39d7c943485a5ed92607bb8556"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.268337 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "50c39a8e067c5fffad46620e403934a2"
server_key_iv: "68ab3b39d7c943485a5ed92607bb8556"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.272921 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:49.276099 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:49.277449 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:12:49.277599 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "50c39a8e067c5fffad46620e403934a2"
server_key_iv: "68ab3b39d7c943485a5ed92607bb8556"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.277805 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:49.292670 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:49.296299 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:49.296576 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:49.299069 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:49.299188 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:49.299347 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:49.299430 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:49.321484 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.299392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56817 (local address 127.0.97.62:39731)
0927 05:12:49.299579 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.299586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.300891 (+ 1305us) server_negotiation.cc:407] Connection header received
0927 05:12:49.303019 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.303027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.303132 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.303379 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:49.305358 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.306686 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.309254 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.309659 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.309822 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.315138 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:49.315145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:49.315151 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:49.315192 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:49.318578 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.319299 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.319307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.319332 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.319478 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.319911 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.319916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.319919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.320183 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:49.320341 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.320950 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.321171 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:12:49.322402 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.299870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.2:56817)
0927 05:12:49.300713 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.300769 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.302695 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.303611 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.303619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.304382 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:49.305142 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.305172 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.306861 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.306868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.309081 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.309095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.309926 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.309940 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.310285 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.311030 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:49.311068 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:49.314868 (+ 3800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:49.318708 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.318716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.318730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.319178 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.319595 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.319600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.319604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.319792 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.320314 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:49.320322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:49.320616 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.320847 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.321315 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":383,"threads_started":1}
I20260927 05:12:49.343922 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.328520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.2:58661)
0927 05:12:49.329166 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.329189 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.329281 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.329618 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.329620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.330025 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:49.330275 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.330283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.331945 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.331952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.334115 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.334131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.334962 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.334970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.335251 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.335917 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:49.335948 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:49.337179 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:49.340734 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.340738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.340773 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.341487 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.341991 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.341996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.342002 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.342178 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.342991 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:49.342998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:49.343264 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.343394 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.343600 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":387,"threads_started":1}
I20260927 05:12:49.344103 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.328511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58661 (local address 127.0.97.62:39731)
0927 05:12:49.328693 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.328700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.329264 (+ 564us) server_negotiation.cc:407] Connection header received
0927 05:12:49.329383 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.329391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.329479 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.329592 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:49.330402 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.331744 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.334266 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.334759 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.334901 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.337468 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:49.337472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:49.337475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:49.337502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:49.340598 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.341625 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.341633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.341637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.341857 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.342346 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.342353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.342359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.342797 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:49.342938 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.343528 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.343753 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:12:49.357045 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45709
I20260927 05:12:49.357079 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45709 every 8 connection(s)
I20260927 05:12:49.358294 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:12:49.360261 2478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39731
I20260927 05:12:49.360438 2478 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:49.360972 2478 heartbeater.cc:511] Master 127.0.97.62:39731 requested a full tablet report, sending...
I20260927 05:12:49.362172 2132 ts_manager.cc:194] Registered new tserver with Master: bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709)
I20260927 05:12:49.362869 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56817
I20260927 05:12:49.363796 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2343
I20260927 05:12:49.363970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:12:49.364382 388 external_mini_cluster.cc:1428] Setting key 7ae9b0a42c5675d5876c48246a131e88
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:12:49.399523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39731
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:49.709486 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:49.709779 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:49.709964 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:49.716949 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:49.717083 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:49.717216 2482 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:12:49.717303 2482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:12:49.717438 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:12:49.726840 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:49.728026 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:49.729293 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:49.742092 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:49.742131 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:49.742167 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:49.743068 2482 server_base.cc:1034] running on GCE node
I20260927 05:12:49.743680 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:49.744547 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:49.745738 2482 hybrid_clock.cc:648] HybridClock initialized: now 1790485969745679 us; error 57 us; skew 500 ppm
I20260927 05:12:49.746039 2482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:49.748968 2482 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:12:49.750386 2482 webserver.cc:460] Webserver started at http://127.0.97.3:46841/ using document root <none> and password file <none>
I20260927 05:12:49.750979 2482 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:49.751149 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:49.751406 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:49.754196 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "b6e41bbd8a884a52a08330fe715d7312"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "659605f9ffc27100a915b49a7be45089"
server_key_iv: "e222b112f8709f89243039e6941ff509"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.754925 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b6e41bbd8a884a52a08330fe715d7312"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "659605f9ffc27100a915b49a7be45089"
server_key_iv: "e222b112f8709f89243039e6941ff509"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.759811 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:12:49.762923 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:49.764192 2482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:49.764365 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "b6e41bbd8a884a52a08330fe715d7312"
format_stamp: "Formatted at 2026-09-27 05:12:49 on dist-test-slave-1t6p"
server_key: "659605f9ffc27100a915b49a7be45089"
server_key_iv: "e222b112f8709f89243039e6941ff509"
server_key_version: "encryptionkey@0"
I20260927 05:12:49.764544 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:49.780159 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:49.784278 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:49.784545 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:49.786888 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:49.786972 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:49.787143 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:49.787236 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:49.809207 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.787582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34673 (local address 127.0.97.62:39731)
0927 05:12:49.787754 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.787760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.789190 (+ 1430us) server_negotiation.cc:407] Connection header received
0927 05:12:49.791397 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.791404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.791485 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.791567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:49.793735 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.795115 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.796651 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.797397 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.797532 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.802264 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:49.802273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:49.802282 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:49.802310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:49.805274 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.806185 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.806192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.806196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.806367 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.806918 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.806925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.806931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.807362 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:49.807523 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.808459 (+ 936us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.808736 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:12:49.810704 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.788145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.3:34673)
0927 05:12:49.789008 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.789058 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.791046 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.792093 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.792104 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.792776 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:49.793562 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.793579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.795279 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.795283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.796508 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.796519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.797879 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.797888 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.798220 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.798948 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:49.798971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:49.802058 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:49.805431 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.805446 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.805466 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.806061 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.806506 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.806513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.806519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.806793 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.807514 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:49.807521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:49.808006 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.808340 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.808958 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":402,"threads_started":1}
I20260927 05:12:49.831795 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.816454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.97.3:58387)
0927 05:12:49.819518 (+ 3064us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.819541 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.819701 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.820085 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.820090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.820507 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:49.820754 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.820762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.822125 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.822130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.823594 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.823604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.824365 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.824371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.824557 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.825249 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:49.825266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:49.825982 (+ 716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:49.828877 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.828883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.828887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.829256 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.829636 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:49.829641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:49.829644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:49.829770 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:49.830353 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:49.830357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:49.830732 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.830940 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.831291 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2880,"thread_start_us":2670,"threads_started":1}
I20260927 05:12:49.832463 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.816334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58387 (local address 127.0.97.62:39731)
0927 05:12:49.816509 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.816517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.819642 (+ 3125us) server_negotiation.cc:407] Connection header received
0927 05:12:49.819883 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.819895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.819958 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.820057 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:49.820886 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.821986 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.823723 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.824182 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.824304 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.826155 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:49.826161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:49.826165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:49.826201 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:49.828713 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.829397 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.829404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.829408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.829528 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.829897 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.829904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.829909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.830246 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:49.830381 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.832103 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.832245 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:12:49.842269 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35999
I20260927 05:12:49.842296 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35999 every 8 connection(s)
I20260927 05:12:49.843452 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:12:49.844477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2482
I20260927 05:12:49.844659 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:12:49.845173 388 external_mini_cluster.cc:1428] Setting key 4fbc2fd3d5e85b2a833f9eb051ce7aa3
I20260927 05:12:49.845428 2617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39731
I20260927 05:12:49.845603 2617 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:49.846382 2617 heartbeater.cc:511] Master 127.0.97.62:39731 requested a full tablet report, sending...
I20260927 05:12:49.847903 2131 ts_manager.cc:194] Registered new tserver with Master: b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:49.848758 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34673
I20260927 05:12:49.860363 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:12:49.876884 2339 heartbeater.cc:503] Master 127.0.97.62:39731 was elected leader, sending a full tablet report...
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:12:49 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:49.904582 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.891201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.97.62:39731)
0927 05:12:49.891438 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.891445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.891648 (+ 203us) server_negotiation.cc:407] Connection header received
0927 05:12:49.891867 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.891872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.891931 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.892035 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:49.893361 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.894091 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.895388 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.895834 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.895967 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.899659 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:49.899664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:49.899668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:49.899697 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:49.902028 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.902626 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.902636 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.902638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.902742 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:49.903133 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:49.903138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:49.903148 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:49.903401 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:49.903647 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.903949 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.904171 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:12:49.908571 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:49.916937 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:49.952842 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.929055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.97.2:45709)
0927 05:12:49.945257 (+ 16202us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.945265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.945291 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:12:49.945402 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.945413 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.945730 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.946605 (+ 875us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:49.946787 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.947902 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.950723 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.951773 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.951932 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.952155 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.952389 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.952551 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16050,"thread_start_us":15930,"threads_started":1}
I20260927 05:12:49.952967 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.928936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.0.1:59452)
0927 05:12:49.937170 (+ 8234us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.937210 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.937375 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.945954 (+ 8579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.945961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.945993 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:49.946417 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.946431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.948036 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.948042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.950567 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.950584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.951959 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.951964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.952139 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.952195 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.952343 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.952551 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7987,"mutex_wait_us":150,"thread_start_us":15232,"threads_started":1}
I20260927 05:12:49.953811 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.930147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.0.97.1:37569)
0927 05:12:49.937925 (+ 7778us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.937936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.944140 (+ 6204us) server_negotiation.cc:407] Connection header received
0927 05:12:49.944713 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.944725 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.945053 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.945232 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:49.946107 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.947668 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.950682 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.952054 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.952156 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.952355 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.953364 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.953533 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7625,"thread_start_us":7494,"threads_started":1}
I20260927 05:12:49.954015 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.929671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.0.1:54804)
0927 05:12:49.945954 (+ 16283us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.945972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.946154 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.947058 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.947062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.947086 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:49.947422 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.947432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.949004 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.949016 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.950977 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.950991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.953212 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.953220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.953405 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.953427 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.953525 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.953644 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16158,"mutex_wait_us":2,"thread_start_us":7100,"threads_started":1}
I20260927 05:12:49.954049 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.930827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.0.97.3:35999)
0927 05:12:49.944655 (+ 13828us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:49.944667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:49.946054 (+ 1387us) server_negotiation.cc:407] Connection header received
0927 05:12:49.946352 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:49.946361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:49.946559 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:49.946721 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:49.947627 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.948827 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.951161 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.952977 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.953083 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.953295 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:49.953590 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 05:12:49.953738 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13601,"thread_start_us":13838,"threads_started":1}
I20260927 05:12:49.954834 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:49.929099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37569 (local address 127.0.0.1:47946)
0927 05:12:49.943993 (+ 14894us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:49.944017 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:49.944467 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:49.945314 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:49.945321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:49.945351 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:49.945881 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.945895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.947789 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.947796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.950486 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:49.950502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:49.952547 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:49.952553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:49.952742 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:49.952775 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:12:49.952863 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:12:49.954503 (+ 1640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14734,"mutex_wait_us":45,"thread_start_us":16561,"threads_started":1}
I20260927 05:12:49.957686 2547 tablet_service.cc:1467] Processing CreateTablet for tablet a2fff607baa64251a0c1df7d11cfb180 (DEFAULT_TABLE table=test-table [id=a65d3a547688416e9e88e6249265a421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:49.957686 2408 tablet_service.cc:1467] Processing CreateTablet for tablet a2fff607baa64251a0c1df7d11cfb180 (DEFAULT_TABLE table=test-table [id=a65d3a547688416e9e88e6249265a421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:49.957686 2269 tablet_service.cc:1467] Processing CreateTablet for tablet a2fff607baa64251a0c1df7d11cfb180 (DEFAULT_TABLE table=test-table [id=a65d3a547688416e9e88e6249265a421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:49.958974 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fff607baa64251a0c1df7d11cfb180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:49.959057 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fff607baa64251a0c1df7d11cfb180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:49.959059 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fff607baa64251a0c1df7d11cfb180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:49.969823 2636 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Bootstrap starting.
I20260927 05:12:49.972980 2636 tablet_bootstrap.cc:654] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:49.972966 2638 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Bootstrap starting.
I20260927 05:12:49.974323 2636 log.cc:824] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:49.975657 2637 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Bootstrap starting.
I20260927 05:12:49.976567 2638 tablet_bootstrap.cc:654] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:49.978011 2638 log.cc:824] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:49.978478 2637 tablet_bootstrap.cc:654] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:49.979463 2637 log.cc:824] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:49.982064 2637 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: No bootstrap required, opened a new log
I20260927 05:12:49.982358 2637 ts_tablet_manager.cc:1397] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 05:12:49.984459 2636 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: No bootstrap required, opened a new log
I20260927 05:12:49.984783 2636 ts_tablet_manager.cc:1397] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260927 05:12:49.984966 2638 tablet_bootstrap.cc:492] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: No bootstrap required, opened a new log
I20260927 05:12:49.985213 2638 ts_tablet_manager.cc:1397] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260927 05:12:49.992314 2638 raft_consensus.cc:348] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.992743 2638 raft_consensus.cc:374] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:49.992856 2638 raft_consensus.cc:729] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Initialized, Role: FOLLOWER
I20260927 05:12:49.992619 2637 raft_consensus.cc:348] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.993007 2637 raft_consensus.cc:374] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:49.993103 2637 raft_consensus.cc:729] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Initialized, Role: FOLLOWER
I20260927 05:12:49.993583 2638 consensus_queue.cc:260] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.993686 2637 consensus_queue.cc:260] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.995249 2636 raft_consensus.cc:348] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.995683 2636 raft_consensus.cc:374] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:49.995781 2636 raft_consensus.cc:729] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Initialized, Role: FOLLOWER
I20260927 05:12:49.996198 2478 heartbeater.cc:503] Master 127.0.97.62:39731 was elected leader, sending a full tablet report...
I20260927 05:12:49.996438 2636 consensus_queue.cc:260] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:49.998111 2636 ts_tablet_manager.cc:1428] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 05:12:49.998104 2637 ts_tablet_manager.cc:1428] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 05:12:50.000280 2638 ts_tablet_manager.cc:1428] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260927 05:12:50.000711 2617 heartbeater.cc:503] Master 127.0.97.62:39731 was elected leader, sending a full tablet report...
W20260927 05:12:50.095225 2618 tablet.cc:2369] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:12:50.110225 2479 tablet.cc:2369] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:12:50.120628 2340 tablet.cc:2369] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:12:50.134222 2643 raft_consensus.cc:482] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:50.134498 2643 raft_consensus.cc:504] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.136220 2643 leader_election.cc:288] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:50.143520 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.136750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46665 (local address 127.0.97.3:35999)
0927 05:12:50.136980 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.136987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.137941 (+ 954us) server_negotiation.cc:407] Connection header received
0927 05:12:50.138162 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.138166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.138223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.138318 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.139047 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.139888 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.141752 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.142549 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.142650 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.142724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.143138 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.143271 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 05:12:50.143615 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.136578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36625 (local address 127.0.97.1:37569)
0927 05:12:50.136789 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.136797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.138138 (+ 1341us) server_negotiation.cc:407] Connection header received
0927 05:12:50.138347 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.138356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.138410 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.138504 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.139256 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.139949 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.141889 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.142707 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.142842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.142935 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.143263 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.143399 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:12:50.143612 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.136748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.97.2:46665)
0927 05:12:50.137831 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.137860 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.138007 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.138358 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.138364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.138394 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.138862 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.138872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.140030 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.140036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.141602 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.141613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.142751 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.142759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.142958 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.142989 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.143104 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.143184 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":21,"thread_start_us":892,"threads_started":1}
I20260927 05:12:50.143890 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.136908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37569 (local address 127.0.97.2:36625)
0927 05:12:50.138042 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.138056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.138191 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.138518 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.138526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.138629 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.139115 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.139126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.140078 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.140081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.141733 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.141746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.142916 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.142920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.143101 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.143139 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.143225 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.143333 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":34,"spinlock_wait_cycles":30080,"thread_start_us":451,"threads_started":1}
I20260927 05:12:50.144562 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fff607baa64251a0c1df7d11cfb180" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312" is_pre_election: true
I20260927 05:12:50.144723 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fff607baa64251a0c1df7d11cfb180" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee" is_pre_election: true
I20260927 05:12:50.144979 2571 raft_consensus.cc:2454] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 0.
I20260927 05:12:50.145233 2294 raft_consensus.cc:2454] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 0.
I20260927 05:12:50.145565 2359 leader_election.cc:302] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, bc93e03e00494ddfa1cc11d4acd705bf; no voters:
I20260927 05:12:50.145979 2643 raft_consensus.cc:2781] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:50.146118 2643 raft_consensus.cc:482] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:50.146231 2643 raft_consensus.cc:3037] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.148416 2643 raft_consensus.cc:504] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.149024 2643 leader_election.cc:288] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 election: Requested vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:50.149345 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fff607baa64251a0c1df7d11cfb180" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee"
I20260927 05:12:50.149529 2294 raft_consensus.cc:3037] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.149482 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fff607baa64251a0c1df7d11cfb180" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312"
I20260927 05:12:50.149708 2571 raft_consensus.cc:3037] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.151784 2294 raft_consensus.cc:2454] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 1.
I20260927 05:12:50.151976 2571 raft_consensus.cc:2454] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 1.
I20260927 05:12:50.152153 2361 leader_election.cc:302] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc93e03e00494ddfa1cc11d4acd705bf, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:50.152431 2643 raft_consensus.cc:2781] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:50.152796 2643 raft_consensus.cc:686] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 LEADER]: Becoming Leader. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Running, Role: LEADER
I20260927 05:12:50.153234 2643 consensus_queue.cc:237] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.157670 2130 catalog_manager.cc:5495] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf reported cstate change: term changed from 0 to 1, leader changed from <none> to bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } health_report { overall_health: UNKNOWN } } }
Sep 27 05:12:50 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:50.222092 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.209288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39731 (local address 127.0.0.1:51784)
0927 05:12:50.209817 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.209831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.209946 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.210337 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.210341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.210992 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:50.211329 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.211338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.212298 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.212301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.213391 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.213401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.214048 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.214052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.214154 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.214724 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:50.214743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:50.216877 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:50.219822 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:50.219830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:50.219836 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:50.220210 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:50.220561 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:50.220564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:50.220566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:50.220725 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:50.221210 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:50.221216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:50.221358 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.221498 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.221752 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260927 05:12:50.222119 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.209256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.97.62:39731)
0927 05:12:50.209416 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.209421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.209915 (+ 494us) server_negotiation.cc:407] Connection header received
0927 05:12:50.210110 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.210114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.210186 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.210268 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:50.211458 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.212157 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.213521 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.213900 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.213987 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.217019 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:50.217023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:50.217026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:50.217052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:50.219708 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:50.220330 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:50.220334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:50.220337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:50.220460 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:50.220840 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:50.220844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:50.220846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:50.221078 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:50.221208 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.221605 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.221878 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":57}
I20260927 05:12:50.226969 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:12:50.228060 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:50.234086 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 36bbd25ffe4741cda28b1b40873d6698 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6e649f08448ae8541e085d81c5747]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:50.234222 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 36bbd25ffe4741cda28b1b40873d6698 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6e649f08448ae8541e085d81c5747]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:50.234480 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 36bbd25ffe4741cda28b1b40873d6698 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6e649f08448ae8541e085d81c5747]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:12:50.235074 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bbd25ffe4741cda28b1b40873d6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.235092 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bbd25ffe4741cda28b1b40873d6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.235265 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bbd25ffe4741cda28b1b40873d6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.240752 2637 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: Bootstrap starting.
I20260927 05:12:50.241981 2636 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: Bootstrap starting.
I20260927 05:12:50.242137 2637 tablet_bootstrap.cc:654] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.242580 2638 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: Bootstrap starting.
I20260927 05:12:50.243577 2636 tablet_bootstrap.cc:654] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.244096 2638 tablet_bootstrap.cc:654] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.244750 2637 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: No bootstrap required, opened a new log
I20260927 05:12:50.244899 2637 ts_tablet_manager.cc:1397] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 05:12:50.245807 2636 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: No bootstrap required, opened a new log
I20260927 05:12:50.245725 2637 raft_consensus.cc:348] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.245934 2636 ts_tablet_manager.cc:1397] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:12:50.245957 2637 raft_consensus.cc:374] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.246042 2637 raft_consensus.cc:729] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.246276 2637 consensus_queue.cc:260] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.246644 2636 raft_consensus.cc:348] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.246693 2643 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.246879 2636 raft_consensus.cc:374] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.246981 2643 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.246970 2636 raft_consensus.cc:729] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.247205 2636 consensus_queue.cc:260] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.247552 2637 ts_tablet_manager.cc:1428] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:12:50.247700 2636 ts_tablet_manager.cc:1428] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:50.248011 2644 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.248310 2644 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.249100 2638 tablet_bootstrap.cc:492] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: No bootstrap required, opened a new log
I20260927 05:12:50.249238 2638 ts_tablet_manager.cc:1397] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 05:12:50.250023 2638 raft_consensus.cc:348] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.250418 2638 raft_consensus.cc:374] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.250528 2638 raft_consensus.cc:729] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.250759 2638 consensus_queue.cc:260] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.251297 2642 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.251637 2642 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.251973 2638 ts_tablet_manager.cc:1428] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 05:12:50.377512 2642 raft_consensus.cc:482] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:50.377705 2642 raft_consensus.cc:504] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.379206 2642 leader_election.cc:288] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709)
I20260927 05:12:50.385174 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.379517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58347 (local address 127.0.97.1:37569)
0927 05:12:50.379712 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.379720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.380187 (+ 467us) server_negotiation.cc:407] Connection header received
0927 05:12:50.380401 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.380405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.380463 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.380559 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.380999 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.381734 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.383397 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.384285 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.384425 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.384518 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.384801 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.384917 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:12:50.385340 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.379432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37569 (local address 127.0.97.3:58347)
0927 05:12:50.380081 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.380098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.380253 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.380574 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.380578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.380601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.380847 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.380855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.381866 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.381872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.383256 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.383267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.384437 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.384443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.384614 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.384636 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.384780 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.384904 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":1,"thread_start_us":532,"threads_started":1}
I20260927 05:12:50.385892 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bbd25ffe4741cda28b1b40873d6698" candidate_uuid: "b6e41bbd8a884a52a08330fe715d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee" is_pre_election: true
I20260927 05:12:50.386159 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.379587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33629 (local address 127.0.97.2:45709)
0927 05:12:50.379829 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.379837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.380289 (+ 452us) server_negotiation.cc:407] Connection header received
0927 05:12:50.380462 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.380467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.380567 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.380659 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.381249 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.382235 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.384005 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.384976 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.385110 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.385225 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.385817 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.385937 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:12:50.386142 2294 raft_consensus.cc:2454] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e41bbd8a884a52a08330fe715d7312 in term 0.
I20260927 05:12:50.386252 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.379432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.97.3:33629)
0927 05:12:50.380192 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.380205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.380344 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.380708 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.380712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.380755 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.381079 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.381106 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.382358 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.382363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.383866 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.383876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.385404 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.385410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.385641 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.385666 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.385776 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.385894 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":113,"thread_start_us":294,"threads_started":1}
I20260927 05:12:50.386628 2500 leader_election.cc:302] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:50.387037 2642 raft_consensus.cc:2781] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:50.387164 2642 raft_consensus.cc:482] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:50.387274 2642 raft_consensus.cc:3037] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.387143 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bbd25ffe4741cda28b1b40873d6698" candidate_uuid: "b6e41bbd8a884a52a08330fe715d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" is_pre_election: true
I20260927 05:12:50.387545 2433 raft_consensus.cc:2454] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e41bbd8a884a52a08330fe715d7312 in term 0.
I20260927 05:12:50.389849 2642 raft_consensus.cc:504] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.390343 2642 leader_election.cc:288] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [CANDIDATE]: Term 1 election: Requested vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709)
I20260927 05:12:50.390726 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bbd25ffe4741cda28b1b40873d6698" candidate_uuid: "b6e41bbd8a884a52a08330fe715d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee"
I20260927 05:12:50.390764 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bbd25ffe4741cda28b1b40873d6698" candidate_uuid: "b6e41bbd8a884a52a08330fe715d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
I20260927 05:12:50.390894 2433 raft_consensus.cc:3037] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.390929 2294 raft_consensus.cc:3037] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.392814 2294 raft_consensus.cc:2454] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6e41bbd8a884a52a08330fe715d7312 in term 1.
I20260927 05:12:50.392871 2433 raft_consensus.cc:2454] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6e41bbd8a884a52a08330fe715d7312 in term 1.
I20260927 05:12:50.393354 2501 leader_election.cc:302] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, bc93e03e00494ddfa1cc11d4acd705bf; no voters:
I20260927 05:12:50.393692 2642 raft_consensus.cc:2781] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:50.394074 2642 raft_consensus.cc:686] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [term 1 LEADER]: Becoming Leader. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Running, Role: LEADER
I20260927 05:12:50.394519 2642 consensus_queue.cc:237] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.397596 2660 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: TxnStatusTablet state changed. Reason: New leader b6e41bbd8a884a52a08330fe715d7312. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.397953 2660 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:50.398603 2662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:12:50.398869 2662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:12:50.398849 2131 catalog_manager.cc:5495] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } health_report { overall_health: HEALTHY } } }
I20260927 05:12:50.436987 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.430982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.0.1:54816)
0927 05:12:50.431230 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.431250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.431405 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.431890 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.431893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.431903 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:50.432044 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.432051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.433428 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.433432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.434361 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.434372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.435393 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.435398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.435560 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.435611 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.436568 (+ 957us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.436582 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.436639 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.436754 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:12:50.437057 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.430984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.0.97.3:35999)
0927 05:12:50.431219 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.431225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.431331 (+ 106us) server_negotiation.cc:407] Connection header received
0927 05:12:50.431596 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.431602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.431722 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.431865 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.432164 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.433301 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.434494 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.435227 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.435377 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.435805 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.436460 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.436611 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.436672 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.436751 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 05:12:50.444403 2294 raft_consensus.cc:1261] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Refusing update from remote peer b6e41bbd8a884a52a08330fe715d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.444456 2433 raft_consensus.cc:1261] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Refusing update from remote peer b6e41bbd8a884a52a08330fe715d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.445261 2660 consensus_queue.cc:1035] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.445539 2642 consensus_queue.cc:1035] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.452913 2644 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: TxnStatusTablet state changed. Reason: New leader b6e41bbd8a884a52a08330fe715d7312. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.453217 2644 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.455840 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333830534931374080 to 7333830534739718144. Current Snapshot: MvccSnapshot[applied={T|T < 7333830534931374080}]
I20260927 05:12:50.456844 2642 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.457144 2642 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:50.463640 2643 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: TxnStatusTablet state changed. Reason: New leader b6e41bbd8a884a52a08330fe715d7312. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.463959 2643 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.466661 2660 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.466899 2660 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P b6e41bbd8a884a52a08330fe715d7312: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:12:50.468612 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333830534931374080 to 7333830534739718144. Current Snapshot: MvccSnapshot[applied={T|T < 7333830534931374080}]
I20260927 05:12:50.476224 2643 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.476547 2643 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P bc93e03e00494ddfa1cc11d4acd705bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.478883 2644 tablet_replica.cc:439] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6e41bbd8a884a52a08330fe715d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } } }
I20260927 05:12:50.479204 2644 tablet_replica.cc:442] T 36bbd25ffe4741cda28b1b40873d6698 P edc9de6f53254bfea076e220b4a4edee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:12:50.486140 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.480252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.97.2:45709)
0927 05:12:50.480410 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.480416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.480947 (+ 531us) server_negotiation.cc:407] Connection header received
0927 05:12:50.481096 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.481101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.481203 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.481347 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.481755 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.482473 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.484089 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.484543 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.484694 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.485086 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.485564 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.485736 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.485791 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.485879 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:12:50.501274 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.495938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55235 (local address 127.0.97.3:35999)
0927 05:12:50.496114 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.496118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.496589 (+ 471us) server_negotiation.cc:407] Connection header received
0927 05:12:50.496797 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.496801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.496863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.496967 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.497426 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.498119 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.499551 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.499988 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.500126 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.500560 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.500663 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.500760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.500906 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.500981 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:12:50.501376 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.495947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.97.2:55235)
0927 05:12:50.496491 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.496505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.496661 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.496976 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.496981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.496993 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:50.497276 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.497284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.498257 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.498263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.499415 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.499426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.500171 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.500180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.500301 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.500351 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.500773 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.500785 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.500885 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.501008 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
I20260927 05:12:50.510783 2294 raft_consensus.cc:1261] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Refusing update from remote peer bc93e03e00494ddfa1cc11d4acd705bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.511061 2571 raft_consensus.cc:1261] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Refusing update from remote peer bc93e03e00494ddfa1cc11d4acd705bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.511461 2647 consensus_queue.cc:1035] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.511811 2643 consensus_queue.cc:1035] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.517434 2666 mvcc.cc:205] Tried to move back new op lower bound from 7333830535206461440 to 7333830533750988800. Current Snapshot: MvccSnapshot[applied={T|T < 7333830535206461440}]
I20260927 05:12:50.547503 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.540777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44689 (local address 127.0.97.2:45709)
0927 05:12:50.540943 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.540949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.541604 (+ 655us) server_negotiation.cc:407] Connection header received
0927 05:12:50.541804 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.541809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.541890 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.541978 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.542627 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.543444 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.545522 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.546194 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.546290 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.546900 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.546992 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.547117 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.547247 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.547305 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 05:12:50.547755 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.540789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.97.3:44689)
0927 05:12:50.541507 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.541523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.541670 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.542017 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.542022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.542041 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:50.542487 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.542500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.543572 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.543580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.545364 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.545378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.546470 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.546476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.546652 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.546721 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.547121 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.547132 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.547223 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.547357 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":393,"threads_started":1}
I20260927 05:12:50.594910 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:50.595247 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:50.599203 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Sending DeleteTablet for 3 replicas of tablet a2fff607baa64251a0c1df7d11cfb180
I20260927 05:12:50.600225 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fff607baa64251a0c1df7d11cfb180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260927 05:12:50.600695 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fff607baa64251a0c1df7d11cfb180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260927 05:12:50.601339 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fff607baa64251a0c1df7d11cfb180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54804
I20260927 05:12:50.606230 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:50.606860 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:50.610529 2688 tablet_replica.cc:330] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: stopping tablet replica
I20260927 05:12:50.616766 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (DEFAULT_TABLE table=test-table [id=41607c18df8847ad8d312db31cb01e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:50.616843 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (DEFAULT_TABLE table=test-table [id=41607c18df8847ad8d312db31cb01e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:50.617195 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.617424 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.619791 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (DEFAULT_TABLE table=test-table [id=41607c18df8847ad8d312db31cb01e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:50.620352 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:50.625478 2638 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Bootstrap starting.
I20260927 05:12:50.627148 2638 tablet_bootstrap.cc:654] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.631079 2636 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Bootstrap starting.
I20260927 05:12:50.632840 2636 tablet_bootstrap.cc:654] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.634186 2637 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Bootstrap starting.
I20260927 05:12:50.636426 2637 tablet_bootstrap.cc:654] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:50.645557 2638 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: No bootstrap required, opened a new log
I20260927 05:12:50.645754 2638 ts_tablet_manager.cc:1397] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260927 05:12:50.646554 2638 raft_consensus.cc:348] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.646809 2638 raft_consensus.cc:374] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.646893 2638 raft_consensus.cc:729] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.647183 2638 consensus_queue.cc:260] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.647790 2638 ts_tablet_manager.cc:1428] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 05:12:50.649309 2687 tablet_replica.cc:330] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: stopping tablet replica
I20260927 05:12:50.649772 2687 raft_consensus.cc:2229] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:50.650132 2687 raft_consensus.cc:2258] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:50.652283 2687 ts_tablet_manager.cc:1905] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:50.653323 2637 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: No bootstrap required, opened a new log
I20260927 05:12:50.653549 2637 ts_tablet_manager.cc:1397] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.008s
I20260927 05:12:50.654465 2637 raft_consensus.cc:348] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.654800 2637 raft_consensus.cc:374] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.654937 2637 raft_consensus.cc:729] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.655345 2637 consensus_queue.cc:260] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.656250 2636 tablet_bootstrap.cc:492] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: No bootstrap required, opened a new log
I20260927 05:12:50.656431 2636 ts_tablet_manager.cc:1397] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.003s
I20260927 05:12:50.657230 2636 raft_consensus.cc:348] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.657490 2636 raft_consensus.cc:374] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:50.657603 2636 raft_consensus.cc:729] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Initialized, Role: FOLLOWER
I20260927 05:12:50.657927 2636 consensus_queue.cc:260] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.658736 2637 ts_tablet_manager.cc:1428] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260927 05:12:50.660020 2636 ts_tablet_manager.cc:1428] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:50.661106 2687 ts_tablet_manager.cc:1918] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:50.661258 2687 log.cc:1196] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/a2fff607baa64251a0c1df7d11cfb180
I20260927 05:12:50.661806 2687 ts_tablet_manager.cc:1939] T a2fff607baa64251a0c1df7d11cfb180 P edc9de6f53254bfea076e220b4a4edee: Deleting consensus metadata
I20260927 05:12:50.662899 2688 raft_consensus.cc:2229] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:50.663409 2688 raft_consensus.cc:2258] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:50.663515 2118 catalog_manager.cc:4841] TS edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569): tablet a2fff607baa64251a0c1df7d11cfb180 (table test-table [id=a65d3a547688416e9e88e6249265a421]) successfully deleted
I20260927 05:12:50.664281 2686 tablet_replica.cc:330] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: stopping tablet replica
I20260927 05:12:50.664688 2686 raft_consensus.cc:2229] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:50.665221 2686 raft_consensus.cc:2258] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:50.665697 2688 ts_tablet_manager.cc:1905] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:50.668112 2686 ts_tablet_manager.cc:1905] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:50.674150 2688 ts_tablet_manager.cc:1918] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:50.674386 2688 log.cc:1196] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/a2fff607baa64251a0c1df7d11cfb180
I20260927 05:12:50.675048 2688 ts_tablet_manager.cc:1939] T a2fff607baa64251a0c1df7d11cfb180 P b6e41bbd8a884a52a08330fe715d7312: Deleting consensus metadata
I20260927 05:12:50.676411 2686 ts_tablet_manager.cc:1918] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:50.676613 2686 log.cc:1196] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/a2fff607baa64251a0c1df7d11cfb180
I20260927 05:12:50.677059 2686 ts_tablet_manager.cc:1939] T a2fff607baa64251a0c1df7d11cfb180 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting consensus metadata
I20260927 05:12:50.677098 2116 catalog_manager.cc:4841] TS b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999): tablet a2fff607baa64251a0c1df7d11cfb180 (table test-table [id=a65d3a547688416e9e88e6249265a421]) successfully deleted
I20260927 05:12:50.678459 2119 catalog_manager.cc:4841] TS bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709): tablet a2fff607baa64251a0c1df7d11cfb180 (table test-table [id=a65d3a547688416e9e88e6249265a421]) successfully deleted
I20260927 05:12:50.727619 2644 raft_consensus.cc:482] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:50.727880 2644 raft_consensus.cc:504] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.729735 2644 leader_election.cc:288] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:50.749895 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.730993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43863 (local address 127.0.97.2:45709)
0927 05:12:50.731220 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.731230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.741742 (+ 10512us) server_negotiation.cc:407] Connection header received
0927 05:12:50.742039 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.742045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.742123 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.742187 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.743157 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.744311 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.747338 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.748668 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.748769 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.748877 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.749461 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.749595 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:12:50.750671 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.730607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45755 (local address 127.0.97.3:35999)
0927 05:12:50.730812 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.730820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.741846 (+ 11026us) server_negotiation.cc:407] Connection header received
0927 05:12:50.742128 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.742134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.742236 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.742305 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:12:50.743362 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.744642 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.747864 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.749360 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.749472 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.749593 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.750187 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.750353 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:12:50.751267 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.730065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.97.1:43863)
0927 05:12:50.741639 (+ 11574us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.741660 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.741859 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.742423 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.742430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.742457 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.742886 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.742901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.744503 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.744510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.747122 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.747137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.749066 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.749072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.749247 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.749280 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.750385 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.750537 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11395,"mutex_wait_us":59,"spinlock_wait_cycles":30592,"thread_start_us":10608,"threads_started":1}
I20260927 05:12:50.751461 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.730219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.97.1:45755)
0927 05:12:50.741740 (+ 11521us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.741760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.741966 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.742532 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.742537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.742597 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:12:50.743157 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.743171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.744838 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.744844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.747658 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.747673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.749796 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.749803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.749991 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.750026 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.750855 (+ 829us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.751016 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11388,"mutex_wait_us":37,"spinlock_wait_cycles":1280,"thread_start_us":11023,"threads_started":1}
I20260927 05:12:50.752274 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a858a6fb86d4b2cbf2e8bf3c9672e28" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312" is_pre_election: true
I20260927 05:12:50.752566 2571 raft_consensus.cc:2454] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 0.
I20260927 05:12:50.752652 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a858a6fb86d4b2cbf2e8bf3c9672e28" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" is_pre_election: true
I20260927 05:12:50.752905 2432 raft_consensus.cc:2454] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 0.
I20260927 05:12:50.753157 2219 leader_election.cc:302] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:50.753564 2644 raft_consensus.cc:2781] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:50.753691 2644 raft_consensus.cc:482] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:50.753799 2644 raft_consensus.cc:3037] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.757064 2644 raft_consensus.cc:504] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.758123 2644 leader_election.cc:288] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 election: Requested vote from peers bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:50.758461 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a858a6fb86d4b2cbf2e8bf3c9672e28" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
I20260927 05:12:50.758670 2432 raft_consensus.cc:3037] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.758988 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a858a6fb86d4b2cbf2e8bf3c9672e28" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312"
I20260927 05:12:50.759217 2571 raft_consensus.cc:3037] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:50.761902 2432 raft_consensus.cc:2454] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 1.
I20260927 05:12:50.762421 2222 leader_election.cc:302] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc93e03e00494ddfa1cc11d4acd705bf, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:50.762573 2571 raft_consensus.cc:2454] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 1.
I20260927 05:12:50.762869 2644 raft_consensus.cc:2781] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:50.763478 2644 raft_consensus.cc:686] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 1 LEADER]: Becoming Leader. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Running, Role: LEADER
I20260927 05:12:50.763895 2644 consensus_queue.cc:237] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:50.767702 2130 catalog_manager.cc:5495] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee reported cstate change: term changed from 0 to 1, leader changed from <none> to edc9de6f53254bfea076e220b4a4edee (127.0.97.1). New cstate: current_term: 1 leader_uuid: "edc9de6f53254bfea076e220b4a4edee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } health_report { overall_health: UNKNOWN } } }
I20260927 05:12:50.809473 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.802035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.0.97.1:37569)
0927 05:12:50.802202 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.802208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.802461 (+ 253us) server_negotiation.cc:407] Connection header received
0927 05:12:50.802664 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.802671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.802773 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.802948 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.803380 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.804635 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.806576 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.807233 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.807785 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.808100 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.808845 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.808999 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.809094 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.809172 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:12:50.827762 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.820505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33603 (local address 127.0.97.3:35999)
0927 05:12:50.820670 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.820676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.821354 (+ 678us) server_negotiation.cc:407] Connection header received
0927 05:12:50.821542 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.821549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.821670 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.821772 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.822301 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.823478 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.825657 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.826315 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.826460 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.826956 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.827118 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.827228 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.827384 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.827465 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:12:50.828053 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.820553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35999 (local address 127.0.97.1:33603)
0927 05:12:50.821220 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.821238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.821397 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.821808 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.821813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.821826 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:50.822132 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.822142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.823625 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.823631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.825474 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.825488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.826509 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.826515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.826689 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.826754 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.827232 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.827251 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.827337 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.827468 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":337,"threads_started":1}
I20260927 05:12:50.838169 2432 raft_consensus.cc:1261] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Refusing update from remote peer edc9de6f53254bfea076e220b4a4edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.838227 2571 raft_consensus.cc:1261] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Refusing update from remote peer edc9de6f53254bfea076e220b4a4edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:50.838908 2694 consensus_queue.cc:1035] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.839596 2644 consensus_queue.cc:1035] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:50.888973 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.879156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37569 (local address 127.0.97.3:55387)
0927 05:12:50.879522 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:50.879542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:50.879646 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:50.881665 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:50.881672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:50.881692 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:12:50.881961 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.881979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.883672 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.883677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.885814 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:50.885840 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.887645 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.887651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:50.887866 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.887912 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.888405 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.888415 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:12:50.888483 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 05:12:50.888623 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260927 05:12:50.889665 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:50.879816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55387 (local address 127.0.97.1:37569)
0927 05:12:50.879965 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:50.879971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:50.879985 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:12:50.880039 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:50.880045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:50.880109 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:50.880184 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:50.882390 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.883517 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.886173 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:50.886880 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:50.886990 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:50.888105 (+ 1115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:50.888247 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:50.888319 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:50.889288 (+ 969us) server_negotiation.cc:299] Negotiation successful
0927 05:12:50.889362 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 05:12:50.890929 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 05:12:52.501008 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:52.501222 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:52.504856 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Sending DeleteTablet for 3 replicas of tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28
I20260927 05:12:52.505542 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260927 05:12:52.505661 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260927 05:12:52.505704 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54804
I20260927 05:12:52.518074 2725 tablet_replica.cc:330] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: stopping tablet replica
I20260927 05:12:52.518613 2724 tablet_replica.cc:330] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: stopping tablet replica
I20260927 05:12:52.518863 2726 tablet_replica.cc:330] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: stopping tablet replica
I20260927 05:12:52.518884 2724 raft_consensus.cc:2229] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:52.518901 2725 raft_consensus.cc:2229] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:52.519188 2725 raft_consensus.cc:2258] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.519177 2726 raft_consensus.cc:2229] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:52.519433 2726 raft_consensus.cc:2258] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.519498 2724 raft_consensus.cc:2258] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.520243 2725 ts_tablet_manager.cc:1905] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.520325 2726 ts_tablet_manager.cc:1905] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.520896 2724 ts_tablet_manager.cc:1905] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.526842 2725 ts_tablet_manager.cc:1918] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:52.526842 2726 ts_tablet_manager.cc:1918] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:52.526963 2726 log.cc:1196] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/2a858a6fb86d4b2cbf2e8bf3c9672e28
I20260927 05:12:52.527058 2725 log.cc:1196] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/2a858a6fb86d4b2cbf2e8bf3c9672e28
I20260927 05:12:52.527458 2726 ts_tablet_manager.cc:1939] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P b6e41bbd8a884a52a08330fe715d7312: Deleting consensus metadata
I20260927 05:12:52.527578 2725 ts_tablet_manager.cc:1939] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting consensus metadata
I20260927 05:12:52.528517 2116 catalog_manager.cc:4841] TS b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999): tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (table test-table [id=41607c18df8847ad8d312db31cb01e88]) successfully deleted
I20260927 05:12:52.528517 2119 catalog_manager.cc:4841] TS bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709): tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (table test-table [id=41607c18df8847ad8d312db31cb01e88]) successfully deleted
I20260927 05:12:52.528630 2724 ts_tablet_manager.cc:1918] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:52.528743 2724 log.cc:1196] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/2a858a6fb86d4b2cbf2e8bf3c9672e28
I20260927 05:12:52.529153 2724 ts_tablet_manager.cc:1939] T 2a858a6fb86d4b2cbf2e8bf3c9672e28 P edc9de6f53254bfea076e220b4a4edee: Deleting consensus metadata
I20260927 05:12:52.529989 2118 catalog_manager.cc:4841] TS edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569): tablet 2a858a6fb86d4b2cbf2e8bf3c9672e28 (table test-table [id=41607c18df8847ad8d312db31cb01e88]) successfully deleted
Sep 27 05:12:52 dist-test-slave-1t6p krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:12:52 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:52.560848 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:52.547362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.0.97.62:39731)
0927 05:12:52.547875 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:52.547879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:52.547939 (+ 60us) server_negotiation.cc:407] Connection header received
0927 05:12:52.548077 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:52.548081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:52.548138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:52.548227 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:52.549413 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:52.550255 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:52.551506 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:52.551891 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:52.551982 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:52.555499 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:52.555504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:52.555507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:52.555536 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:52.558455 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:52.559043 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:52.559050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:52.559053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:52.559165 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:52.559506 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:52.559511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:52.559515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:52.559770 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:52.560004 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:52.560289 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 05:12:52.560497 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":279,"threads_started":1}
I20260927 05:12:52.564021 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:52.564589 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:52.570453 2269 tablet_service.cc:1467] Processing CreateTablet for tablet cc52bef557b84fb2b79be8a15194ba23 (DEFAULT_TABLE table=test-table [id=533f84497bac45df86d6b8c5c4d2eb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.570546 2408 tablet_service.cc:1467] Processing CreateTablet for tablet cc52bef557b84fb2b79be8a15194ba23 (DEFAULT_TABLE table=test-table [id=533f84497bac45df86d6b8c5c4d2eb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.570888 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc52bef557b84fb2b79be8a15194ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.571043 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc52bef557b84fb2b79be8a15194ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.575700 2735 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Bootstrap starting.
I20260927 05:12:52.575909 2547 tablet_service.cc:1467] Processing CreateTablet for tablet cc52bef557b84fb2b79be8a15194ba23 (DEFAULT_TABLE table=test-table [id=533f84497bac45df86d6b8c5c4d2eb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.576254 2736 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Bootstrap starting.
I20260927 05:12:52.576309 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc52bef557b84fb2b79be8a15194ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.577665 2736 tablet_bootstrap.cc:654] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.577708 2735 tablet_bootstrap.cc:654] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.580724 2735 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: No bootstrap required, opened a new log
I20260927 05:12:52.580868 2735 ts_tablet_manager.cc:1397] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 05:12:52.581461 2736 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: No bootstrap required, opened a new log
I20260927 05:12:52.581588 2736 ts_tablet_manager.cc:1397] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 05:12:52.581619 2735 raft_consensus.cc:348] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.581877 2735 raft_consensus.cc:374] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.581967 2735 raft_consensus.cc:729] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.582187 2735 consensus_queue.cc:260] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.582332 2736 raft_consensus.cc:348] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.582643 2736 raft_consensus.cc:374] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.582764 2736 raft_consensus.cc:729] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.582993 2735 ts_tablet_manager.cc:1428] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 05:12:52.583071 2736 consensus_queue.cc:260] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.584187 2736 ts_tablet_manager.cc:1428] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:52.591135 2740 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Bootstrap starting.
I20260927 05:12:52.592729 2740 tablet_bootstrap.cc:654] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.594484 2740 tablet_bootstrap.cc:492] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: No bootstrap required, opened a new log
I20260927 05:12:52.594615 2740 ts_tablet_manager.cc:1397] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:12:52.595261 2740 raft_consensus.cc:348] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.595448 2740 raft_consensus.cc:374] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.595566 2740 raft_consensus.cc:729] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.595775 2740 consensus_queue.cc:260] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.596240 2740 ts_tablet_manager.cc:1428] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 05:12:52.681523 2739 raft_consensus.cc:482] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:52.681706 2739 raft_consensus.cc:504] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.682500 2739 leader_election.cc:288] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:52.682958 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc52bef557b84fb2b79be8a15194ba23" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee" is_pre_election: true
I20260927 05:12:52.683097 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc52bef557b84fb2b79be8a15194ba23" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312" is_pre_election: true
I20260927 05:12:52.683269 2293 raft_consensus.cc:2454] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 0.
I20260927 05:12:52.683331 2566 raft_consensus.cc:2454] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 0.
I20260927 05:12:52.683745 2359 leader_election.cc:302] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, bc93e03e00494ddfa1cc11d4acd705bf; no voters:
I20260927 05:12:52.684031 2739 raft_consensus.cc:2781] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:52.684142 2739 raft_consensus.cc:482] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:52.684252 2739 raft_consensus.cc:3037] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:52.686246 2739 raft_consensus.cc:504] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.686750 2739 leader_election.cc:288] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 election: Requested vote from peers edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:52.687209 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc52bef557b84fb2b79be8a15194ba23" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312"
I20260927 05:12:52.687222 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc52bef557b84fb2b79be8a15194ba23" candidate_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc9de6f53254bfea076e220b4a4edee"
I20260927 05:12:52.687440 2293 raft_consensus.cc:3037] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:52.687440 2566 raft_consensus.cc:3037] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:52.690286 2566 raft_consensus.cc:2454] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 1.
I20260927 05:12:52.690286 2293 raft_consensus.cc:2454] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc93e03e00494ddfa1cc11d4acd705bf in term 1.
I20260927 05:12:52.690683 2361 leader_election.cc:302] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc93e03e00494ddfa1cc11d4acd705bf, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:52.690961 2739 raft_consensus.cc:2781] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:52.691416 2739 raft_consensus.cc:686] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 LEADER]: Becoming Leader. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Running, Role: LEADER
I20260927 05:12:52.691689 2739 consensus_queue.cc:237] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.694648 2130 catalog_manager.cc:5495] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf reported cstate change: term changed from 0 to 1, leader changed from <none> to bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } health_report { overall_health: UNKNOWN } } }
I20260927 05:12:52.738749 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:52.734010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.0.97.2:45709)
0927 05:12:52.734548 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:52.734552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:52.734570 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:12:52.734614 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:52.734617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:52.734669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:52.734749 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:52.735124 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:52.735850 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:52.737050 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:52.737511 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:52.737619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:52.738037 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:52.738126 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:52.738194 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:52.738322 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 05:12:52.738377 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":337,"threads_started":1}
I20260927 05:12:52.746380 2293 raft_consensus.cc:1261] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Refusing update from remote peer bc93e03e00494ddfa1cc11d4acd705bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:52.746380 2566 raft_consensus.cc:1261] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Refusing update from remote peer bc93e03e00494ddfa1cc11d4acd705bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:52.746910 2742 consensus_queue.cc:1035] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:52.747216 2739 consensus_queue.cc:1035] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:52.786207 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:52.786408 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:52.788996 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Sending DeleteTablet for 3 replicas of tablet cc52bef557b84fb2b79be8a15194ba23
I20260927 05:12:52.789665 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet cc52bef557b84fb2b79be8a15194ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260927 05:12:52.789796 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet cc52bef557b84fb2b79be8a15194ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54804
I20260927 05:12:52.789781 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet cc52bef557b84fb2b79be8a15194ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260927 05:12:52.790045 2725 tablet_replica.cc:330] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: stopping tablet replica
I20260927 05:12:52.790064 2726 tablet_replica.cc:330] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: stopping tablet replica
I20260927 05:12:52.790066 2724 tablet_replica.cc:330] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: stopping tablet replica
I20260927 05:12:52.790237 2726 raft_consensus.cc:2229] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:52.790236 2725 raft_consensus.cc:2229] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:52.790277 2724 raft_consensus.cc:2229] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:52.790400 2726 raft_consensus.cc:2258] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.790452 2724 raft_consensus.cc:2258] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.790510 2725 raft_consensus.cc:2258] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:52.791435 2725 ts_tablet_manager.cc:1905] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.791543 2726 ts_tablet_manager.cc:1905] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.791543 2724 ts_tablet_manager.cc:1905] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:52.794324 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:12:52.794868 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:12:52.796679 2725 ts_tablet_manager.cc:1918] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:52.796829 2725 log.cc:1196] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/cc52bef557b84fb2b79be8a15194ba23
I20260927 05:12:52.797206 2725 ts_tablet_manager.cc:1939] T cc52bef557b84fb2b79be8a15194ba23 P bc93e03e00494ddfa1cc11d4acd705bf: Deleting consensus metadata
I20260927 05:12:52.797645 2724 ts_tablet_manager.cc:1918] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:52.797798 2724 log.cc:1196] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/cc52bef557b84fb2b79be8a15194ba23
I20260927 05:12:52.798135 2119 catalog_manager.cc:4841] TS bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709): tablet cc52bef557b84fb2b79be8a15194ba23 (table test-table [id=533f84497bac45df86d6b8c5c4d2eb14]) successfully deleted
I20260927 05:12:52.798209 2724 ts_tablet_manager.cc:1939] T cc52bef557b84fb2b79be8a15194ba23 P edc9de6f53254bfea076e220b4a4edee: Deleting consensus metadata
I20260927 05:12:52.798671 2726 ts_tablet_manager.cc:1918] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:12:52.798822 2726 log.cc:1196] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/cc52bef557b84fb2b79be8a15194ba23
I20260927 05:12:52.799139 2118 catalog_manager.cc:4841] TS edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569): tablet cc52bef557b84fb2b79be8a15194ba23 (table test-table [id=533f84497bac45df86d6b8c5c4d2eb14]) successfully deleted
I20260927 05:12:52.799307 2726 ts_tablet_manager.cc:1939] T cc52bef557b84fb2b79be8a15194ba23 P b6e41bbd8a884a52a08330fe715d7312: Deleting consensus metadata
I20260927 05:12:52.800213 2116 catalog_manager.cc:4841] TS b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999): tablet cc52bef557b84fb2b79be8a15194ba23 (table test-table [id=533f84497bac45df86d6b8c5c4d2eb14]) successfully deleted
I20260927 05:12:52.801409 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 45bf024e9cba45dd841a4954eb824f4f (DEFAULT_TABLE table=test-table [id=f2e0aeebd44f43df991c06e2c8a32989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.801687 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 45bf024e9cba45dd841a4954eb824f4f (DEFAULT_TABLE table=test-table [id=f2e0aeebd44f43df991c06e2c8a32989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.801836 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bf024e9cba45dd841a4954eb824f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.801752 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 45bf024e9cba45dd841a4954eb824f4f (DEFAULT_TABLE table=test-table [id=f2e0aeebd44f43df991c06e2c8a32989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:12:52.802049 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bf024e9cba45dd841a4954eb824f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.802099 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bf024e9cba45dd841a4954eb824f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:52.807169 2740 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Bootstrap starting.
I20260927 05:12:52.807734 2736 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Bootstrap starting.
I20260927 05:12:52.808450 2740 tablet_bootstrap.cc:654] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.809051 2736 tablet_bootstrap.cc:654] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.810266 2735 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Bootstrap starting.
I20260927 05:12:52.810492 2740 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: No bootstrap required, opened a new log
I20260927 05:12:52.810657 2740 ts_tablet_manager.cc:1397] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 05:12:52.810825 2736 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: No bootstrap required, opened a new log
I20260927 05:12:52.811347 2736 ts_tablet_manager.cc:1397] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260927 05:12:52.811764 2740 raft_consensus.cc:348] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.811964 2740 raft_consensus.cc:374] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.812057 2740 raft_consensus.cc:729] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e41bbd8a884a52a08330fe715d7312, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.812009 2736 raft_consensus.cc:348] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.812209 2735 tablet_bootstrap.cc:654] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:52.812245 2736 raft_consensus.cc:374] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.812327 2736 raft_consensus.cc:729] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.812552 2736 consensus_queue.cc:260] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.813205 2736 ts_tablet_manager.cc:1428] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 05:12:52.812316 2740 consensus_queue.cc:260] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.814014 2740 ts_tablet_manager.cc:1428] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 05:12:52.820320 2735 tablet_bootstrap.cc:492] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: No bootstrap required, opened a new log
I20260927 05:12:52.820451 2735 ts_tablet_manager.cc:1397] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260927 05:12:52.820933 2735 raft_consensus.cc:348] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.821187 2735 raft_consensus.cc:374] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:52.821236 2735 raft_consensus.cc:729] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc93e03e00494ddfa1cc11d4acd705bf, State: Initialized, Role: FOLLOWER
I20260927 05:12:52.821455 2735 consensus_queue.cc:260] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:52.821899 2735 ts_tablet_manager.cc:1428] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 05:12:53.076366 2714 raft_consensus.cc:482] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:12:53.076550 2714 raft_consensus.cc:504] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:53.077316 2714 leader_election.cc:288] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:53.077867 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bf024e9cba45dd841a4954eb824f4f" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" is_pre_election: true
I20260927 05:12:53.077909 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bf024e9cba45dd841a4954eb824f4f" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312" is_pre_election: true
I20260927 05:12:53.078099 2432 raft_consensus.cc:2454] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 0.
I20260927 05:12:53.078126 2566 raft_consensus.cc:2454] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 0.
I20260927 05:12:53.078532 2219 leader_election.cc:302] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e41bbd8a884a52a08330fe715d7312, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:53.078785 2714 raft_consensus.cc:2781] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:12:53.078903 2714 raft_consensus.cc:482] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:12:53.079012 2714 raft_consensus.cc:3037] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:53.081009 2714 raft_consensus.cc:504] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:53.081600 2714 leader_election.cc:288] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 election: Requested vote from peers bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709), b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999)
I20260927 05:12:53.081987 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bf024e9cba45dd841a4954eb824f4f" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc93e03e00494ddfa1cc11d4acd705bf"
I20260927 05:12:53.082042 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bf024e9cba45dd841a4954eb824f4f" candidate_uuid: "edc9de6f53254bfea076e220b4a4edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e41bbd8a884a52a08330fe715d7312"
I20260927 05:12:53.082170 2432 raft_consensus.cc:3037] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:53.082201 2566 raft_consensus.cc:3037] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:53.084338 2432 raft_consensus.cc:2454] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 1.
I20260927 05:12:53.084748 2222 leader_election.cc:302] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc93e03e00494ddfa1cc11d4acd705bf, edc9de6f53254bfea076e220b4a4edee; no voters:
I20260927 05:12:53.084805 2566 raft_consensus.cc:2454] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc9de6f53254bfea076e220b4a4edee in term 1.
I20260927 05:12:53.085047 2714 raft_consensus.cc:2781] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:53.085440 2714 raft_consensus.cc:686] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 1 LEADER]: Becoming Leader. State: Replica: edc9de6f53254bfea076e220b4a4edee, State: Running, Role: LEADER
I20260927 05:12:53.085726 2714 consensus_queue.cc:237] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } }
I20260927 05:12:53.087867 2130 catalog_manager.cc:5495] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee reported cstate change: term changed from 0 to 1, leader changed from <none> to edc9de6f53254bfea076e220b4a4edee (127.0.97.1). New cstate: current_term: 1 leader_uuid: "edc9de6f53254bfea076e220b4a4edee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc9de6f53254bfea076e220b4a4edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 } health_report { overall_health: UNKNOWN } } }
I20260927 05:12:53.111137 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:53.106436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.97.1:37569)
0927 05:12:53.106929 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:53.106933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:53.106953 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:12:53.107050 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:53.107055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:53.107122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:53.107202 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:12:53.107543 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:53.108268 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:53.109547 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:53.109916 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:53.110005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:53.110386 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:12:53.110476 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:12:53.110540 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:53.110649 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 05:12:53.110731 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260927 05:12:53.119030 2432 raft_consensus.cc:1261] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Refusing update from remote peer edc9de6f53254bfea076e220b4a4edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:53.119023 2566 raft_consensus.cc:1261] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Refusing update from remote peer edc9de6f53254bfea076e220b4a4edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:12:53.119555 2714 consensus_queue.cc:1035] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc93e03e00494ddfa1cc11d4acd705bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:12:53.119845 2753 consensus_queue.cc:1035] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e41bbd8a884a52a08330fe715d7312" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 05:12:53.138706 2707 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 05:12:54.711146 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:54.711360 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:12:54.715126 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4643aeae99494dcc8c80f1ec069bc012: Sending DeleteTablet for 3 replicas of tablet 45bf024e9cba45dd841a4954eb824f4f
I20260927 05:12:54.715821 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bf024e9cba45dd841a4954eb824f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260927 05:12:54.715973 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bf024e9cba45dd841a4954eb824f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260927 05:12:54.715973 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bf024e9cba45dd841a4954eb824f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54804
I20260927 05:12:54.717629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2203
I20260927 05:12:54.719308 2781 tablet_replica.cc:330] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: stopping tablet replica
I20260927 05:12:54.719563 2781 raft_consensus.cc:2229] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:12:54.719987 2781 raft_consensus.cc:2258] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:54.720974 2781 ts_tablet_manager.cc:1905] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:54.729970 2781 ts_tablet_manager.cc:1918] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:54.730136 2781 log.cc:1196] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/45bf024e9cba45dd841a4954eb824f4f
I20260927 05:12:54.730683 2781 ts_tablet_manager.cc:1939] T 45bf024e9cba45dd841a4954eb824f4f P edc9de6f53254bfea076e220b4a4edee: Deleting consensus metadata
I20260927 05:12:54.732004 2118 catalog_manager.cc:4841] TS edc9de6f53254bfea076e220b4a4edee (127.0.97.1:37569): tablet 45bf024e9cba45dd841a4954eb824f4f (table test-table [id=f2e0aeebd44f43df991c06e2c8a32989]) successfully deleted
I20260927 05:12:54.733773 2783 tablet_replica.cc:330] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: stopping tablet replica
I20260927 05:12:54.733986 2783 raft_consensus.cc:2229] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:54.734028 2782 tablet_replica.cc:330] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: stopping tablet replica
I20260927 05:12:54.734241 2782 raft_consensus.cc:2229] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:12:54.734242 2783 raft_consensus.cc:2258] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:54.734469 2782 raft_consensus.cc:2258] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:12:54.735443 2782 ts_tablet_manager.cc:1905] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:54.735735 2783 ts_tablet_manager.cc:1905] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:12:54.740337 2782 ts_tablet_manager.cc:1918] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:54.740494 2782 log.cc:1196] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/45bf024e9cba45dd841a4954eb824f4f
I20260927 05:12:54.740960 2782 ts_tablet_manager.cc:1939] T 45bf024e9cba45dd841a4954eb824f4f P b6e41bbd8a884a52a08330fe715d7312: Deleting consensus metadata
I20260927 05:12:54.741566 2783 ts_tablet_manager.cc:1918] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:12:54.741719 2783 log.cc:1196] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/45bf024e9cba45dd841a4954eb824f4f
I20260927 05:12:54.741905 2116 catalog_manager.cc:4841] TS b6e41bbd8a884a52a08330fe715d7312 (127.0.97.3:35999): tablet 45bf024e9cba45dd841a4954eb824f4f (table test-table [id=f2e0aeebd44f43df991c06e2c8a32989]) successfully deleted
I20260927 05:12:54.742589 2783 ts_tablet_manager.cc:1939] T 45bf024e9cba45dd841a4954eb824f4f P bc93e03e00494ddfa1cc11d4acd705bf: Deleting consensus metadata
I20260927 05:12:54.743491 2119 catalog_manager.cc:4841] TS bc93e03e00494ddfa1cc11d4acd705bf (127.0.97.2:45709): tablet 45bf024e9cba45dd841a4954eb824f4f (table test-table [id=f2e0aeebd44f43df991c06e2c8a32989]) successfully deleted
Sep 27 05:12:54 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:12:54.753233 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:54.719384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.0.97.1:37569)
0927 05:12:54.733020 (+ 13636us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:54.733024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:54.733041 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:12:54.733080 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:54.733083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:54.733130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:54.733255 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:54.734857 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.735878 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.737616 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.738228 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.738370 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:54.748033 (+ 9663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:54.748047 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:54.748064 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:54.748094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:54.750921 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.751545 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.751552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.751554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.751673 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.751987 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.751990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.751992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.752335 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:54.752531 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:54.752834 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:12:54.752981 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13543,"thread_start_us":13466,"threads_started":1}
W20260927 05:12:54.753835 2334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38704
W20260927 05:12:54.754443 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:54.754632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2203
I20260927 05:12:54.776433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2343
Sep 27 05:12:54 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:12:54.790757 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:54.777067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.0.97.2:45709)
0927 05:12:54.777547 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:54.777551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:54.777566 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:12:54.777614 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:54.777617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:54.777669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:54.777783 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:54.778942 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.779729 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.780983 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.781472 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.781612 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:54.785731 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:54.785743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:54.785757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:54.785806 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:54.788565 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.789105 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.789111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.789113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.789217 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.789518 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.789521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.789522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.789835 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:54.790032 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:54.790248 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 05:12:54.790392 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":235,"threads_started":1}
W20260927 05:12:54.791422 2473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47638
W20260927 05:12:54.791950 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:54.792102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2343
I20260927 05:12:54.814035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2482
Sep 27 05:12:54 dist-test-slave-1t6p krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:12:54.828085 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:54.814658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.0.97.3:35999)
0927 05:12:54.815203 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:54.815207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:54.815223 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:12:54.815271 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:54.815275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:54.815328 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:54.815428 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:54.816569 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.817375 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.818622 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:54.819045 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:54.819153 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:54.823220 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:54.823232 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:54.823248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:54.823279 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:54.825747 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.826307 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.826314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.826316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.826454 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:54.826741 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:54.826743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:54.826745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:54.827135 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:54.827365 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:54.827592 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 05:12:54.827734 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
W20260927 05:12:54.828687 2612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52752
W20260927 05:12:54.829347 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:12:54.829514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2482
I20260927 05:12:54.850378 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2100
I20260927 05:12:54.851491 2172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:54.964537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2100
2026-09-27T05:12:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8354 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:55 dist-test-slave-1t6p krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:55 dist-test-slave-1t6p krb5kdc[2791](info): set up 2 sockets
Sep 27 05:12:55 dist-test-slave-1t6p krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260927 05:12:56.075033 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:12:56 dist-test-slave-1t6p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:56.246331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45993
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:56.549196 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:56.549463 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:56.549623 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:56.556419 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:56.556509 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:56.556605 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:56.556700 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:56.556777 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:56.565989 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:56.567154 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:56.568397 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:56.579497 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:56.579501 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:56.579550 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:56.580435 2804 server_base.cc:1034] running on GCE node
I20260927 05:12:56.581152 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:56.582311 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:56.583555 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790485976583515 us; error 42 us; skew 500 ppm
I20260927 05:12:56.583953 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:56 dist-test-slave-1t6p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:56.586936 2804 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:56.588271 2804 webserver.cc:460] Webserver started at http://127.0.97.62:36153/ using document root <none> and password file <none>
I20260927 05:12:56.588884 2804 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:56.588981 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:56.589213 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:56.592017 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "57e372b8d2c245f892454ce9768ac402"
format_stamp: "Formatted at 2026-09-27 05:12:56 on dist-test-slave-1t6p"
server_key: "e6d6d679fb2a62ad87c55a92ff8c864f"
server_key_iv: "6163e3ed823af2504e68e3de5012a199"
server_key_version: "encryptionkey@0"
I20260927 05:12:56.592708 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "57e372b8d2c245f892454ce9768ac402"
format_stamp: "Formatted at 2026-09-27 05:12:56 on dist-test-slave-1t6p"
server_key: "e6d6d679fb2a62ad87c55a92ff8c864f"
server_key_iv: "6163e3ed823af2504e68e3de5012a199"
server_key_version: "encryptionkey@0"
I20260927 05:12:56.597348 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:56.600488 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:56.601715 2804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:12:56.601840 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "57e372b8d2c245f892454ce9768ac402"
format_stamp: "Formatted at 2026-09-27 05:12:56 on dist-test-slave-1t6p"
server_key: "e6d6d679fb2a62ad87c55a92ff8c864f"
server_key_iv: "6163e3ed823af2504e68e3de5012a199"
server_key_version: "encryptionkey@0"
I20260927 05:12:56.601994 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:56.626266 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:56.630399 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:56.630610 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:56.648039 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45993
I20260927 05:12:56.648068 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45993 every 8 connection(s)
I20260927 05:12:56.649286 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:12:56.650051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2804
I20260927 05:12:56.650281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:12:56.650954 388 external_mini_cluster.cc:1428] Setting key ccfcfc53d1004887adef70b8d5a6ac65
I20260927 05:12:56.653458 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:12:56.663301 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402: Bootstrap starting.
Sep 27 05:12:56 dist-test-slave-1t6p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:56.666406 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402: Neither blocks nor log segments found. Creating new log.
I20260927 05:12:56.667512 2871 log.cc:824] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402: Log is configured to *not* fsync() on all Append() calls
I20260927 05:12:56.670027 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402: No bootstrap required, opened a new log
I20260927 05:12:56.671509 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:56.652774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.97.62:45993)
0927 05:12:56.653577 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:56.653593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:56.653639 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:12:56.655285 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:56.655305 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:56.655661 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:56.656051 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:56.657503 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:56.658899 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:56.660118 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:56.660781 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:56.660974 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:56.664741 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:56.664749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:56.664762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:56.664801 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:56.667827 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:56.668473 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:56.668483 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:56.668499 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:56.668640 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:56.669003 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:56.669009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:56.669012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:56.669486 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:56.669778 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:56.670190 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 05:12:56.670564 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":338,"threads_started":1}
I20260927 05:12:56.678393 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260927 05:12:56.678735 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:12:56.678838 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e372b8d2c245f892454ce9768ac402, State: Initialized, Role: FOLLOWER
I20260927 05:12:56.679405 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260927 05:12:56.679628 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:12:56.679751 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:12:56.679924 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:12:56.682274 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260927 05:12:56.682727 2871 leader_election.cc:302] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e372b8d2c245f892454ce9768ac402; no voters:
I20260927 05:12:56.683319 2871 leader_election.cc:288] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:12:56.683418 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:12:56.684109 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [term 1 LEADER]: Becoming Leader. State: Replica: 57e372b8d2c245f892454ce9768ac402, State: Running, Role: LEADER
I20260927 05:12:56.684598 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260927 05:12:56.685247 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:12:56.687309 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e372b8d2c245f892454ce9768ac402. Latest consensus state: current_term: 1 leader_uuid: "57e372b8d2c245f892454ce9768ac402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } } }
I20260927 05:12:56.687994 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:56.688514 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57e372b8d2c245f892454ce9768ac402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e372b8d2c245f892454ce9768ac402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } } }
I20260927 05:12:56.691599 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402 [sys.catalog]: This master's current role is: LEADER
I20260927 05:12:56.692715 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:12:56.697783 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:12:56.705927 2885 catalog_manager.cc:1347] Generated new cluster ID: ee1aea76ba8340ce93db502ab9f97a04
I20260927 05:12:56.706068 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:12:56.722766 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:12:56.724018 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:12:56.730616 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e372b8d2c245f892454ce9768ac402: Generated new TSK 0
I20260927 05:12:56.731393 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:12:56.783640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:57.099490 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:57.099869 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:57.100023 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:57.107059 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:57.107193 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:57.107357 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:12:57.116552 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:57.117792 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:57.119102 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:57.131587 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:57.131656 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:57.131628 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:57.132333 2897 server_base.cc:1034] running on GCE node
I20260927 05:12:57.132933 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:57.133878 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:57.135115 2897 hybrid_clock.cc:648] HybridClock initialized: now 1790485977135062 us; error 46 us; skew 500 ppm
I20260927 05:12:57.135368 2897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:57 dist-test-slave-1t6p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:57.138285 2897 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:12:57.139667 2897 webserver.cc:460] Webserver started at http://127.0.97.1:35893/ using document root <none> and password file <none>
I20260927 05:12:57.140276 2897 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:57.140374 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:57.140611 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:57.143836 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "f8c8e2b6fec64f909cd78a6956b752ef"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "8ec2fcbe5cb47f67e7823837a12c5ba3"
server_key_iv: "3586c3fba037f3becf2502061fbff7f9"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.144533 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f8c8e2b6fec64f909cd78a6956b752ef"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "8ec2fcbe5cb47f67e7823837a12c5ba3"
server_key_iv: "3586c3fba037f3becf2502061fbff7f9"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.149212 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:57.152251 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.153438 2897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 05:12:57.153597 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "f8c8e2b6fec64f909cd78a6956b752ef"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "8ec2fcbe5cb47f67e7823837a12c5ba3"
server_key_iv: "3586c3fba037f3becf2502061fbff7f9"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.153775 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:57.163998 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:57.167594 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:57.167874 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:57.168730 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:57.169958 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:57.170027 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.170166 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:57.170230 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.203042 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35497
I20260927 05:12:57.203145 3024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35497 every 8 connection(s)
I20260927 05:12:57.204377 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:12:57.211428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 2897
I20260927 05:12:57.211617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:12:57.212211 388 external_mini_cluster.cc:1428] Setting key a4e8d694769e554dcda8121d8b067189
Sep 27 05:12:57 dist-test-slave-1t6p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:57.226207 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:57.206923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52717 (local address 127.0.97.62:45993)
0927 05:12:57.207340 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:57.207344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:57.208398 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 05:12:57.210132 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:57.210137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:57.210221 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:57.210343 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:57.211917 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.212666 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.214631 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.215071 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.215164 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:57.219569 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:57.219574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:57.219578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:57.219606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:57.223172 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:57.223820 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:57.223828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:57.223835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:57.223982 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:57.224346 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:57.224354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:57.224360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:57.224725 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:57.224883 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:57.225455 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 05:12:57.225706 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":231,"threads_started":1}
I20260927 05:12:57.226765 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:57.207575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45993 (local address 127.0.97.1:52717)
0927 05:12:57.208241 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:57.208279 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:57.209874 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:57.210571 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:57.210579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:57.211134 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:57.211758 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:57.211775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.212838 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.212842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:57.214496 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:57.214509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.215280 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.215288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:57.215623 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:57.216348 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:57.216380 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:57.219373 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:57.223292 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:57.223298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:57.223309 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:57.223689 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:57.224080 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:57.224083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:57.224085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:57.224238 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:57.224842 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:57.224847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:57.225111 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 05:12:57.225316 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 05:12:57.225646 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":264,"threads_started":1}
I20260927 05:12:57.228241 3025 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45993
I20260927 05:12:57.228546 3025 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:57.229216 3025 heartbeater.cc:511] Master 127.0.97.62:45993 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:12:57.231441 2836 ts_manager.cc:194] Registered new tserver with Master: f8c8e2b6fec64f909cd78a6956b752ef (127.0.97.1:35497)
I20260927 05:12:57.232963 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:12:57.252099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:57.556669 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:57.556967 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:57.557121 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:57.564081 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:57.564234 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:57.564405 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:12:57.573386 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:57.574529 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:57.575914 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:57.588070 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:57.588143 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:57.588143 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:57.589042 3031 server_base.cc:1034] running on GCE node
I20260927 05:12:57.589597 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:57.590386 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:57.591578 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790485977591523 us; error 53 us; skew 500 ppm
I20260927 05:12:57.591831 3031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:57 dist-test-slave-1t6p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:57.595242 3031 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:12:57.596616 3031 webserver.cc:460] Webserver started at http://127.0.97.2:33175/ using document root <none> and password file <none>
I20260927 05:12:57.597189 3031 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:57.597282 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:57.597530 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:57.600229 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "487290dad2b748249268c58be6234fe3"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "1201636aba080661a88a59feda8b98de"
server_key_iv: "880a129d27c287eca5d6aeae3e6408c4"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.600912 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "487290dad2b748249268c58be6234fe3"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "1201636aba080661a88a59feda8b98de"
server_key_iv: "880a129d27c287eca5d6aeae3e6408c4"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.605614 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:57.608837 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.610035 3031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:12:57.610183 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "487290dad2b748249268c58be6234fe3"
format_stamp: "Formatted at 2026-09-27 05:12:57 on dist-test-slave-1t6p"
server_key: "1201636aba080661a88a59feda8b98de"
server_key_iv: "880a129d27c287eca5d6aeae3e6408c4"
server_key_version: "encryptionkey@0"
I20260927 05:12:57.610386 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:57.625602 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:57.629264 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:57.629529 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:57.630367 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:57.631596 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:57.631664 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.631834 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:57.631886 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:57.676126 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35599
I20260927 05:12:57.676147 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35599 every 8 connection(s)
I20260927 05:12:57.677377 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:12:57.680009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3031
I20260927 05:12:57.680186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:12:57.680660 388 external_mini_cluster.cc:1428] Setting key 382b494090222c4b82a073d4f0a1b2f4
Sep 27 05:12:57 dist-test-slave-1t6p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:12:57.696919 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:57.679711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39131 (local address 127.0.97.62:45993)
0927 05:12:57.679932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:57.679938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:57.681255 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 05:12:57.683029 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:57.683034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:57.683125 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:57.683231 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:57.684573 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.685763 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.687629 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.688011 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.688104 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:57.691421 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:57.691428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:57.691433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:57.691472 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:57.694167 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:57.694835 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:57.694842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:57.694845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:57.694969 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:57.695389 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:57.695396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:57.695399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:57.695639 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:57.695785 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:57.696388 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 05:12:57.696591 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 05:12:57.697870 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:57.680283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45993 (local address 127.0.97.2:39131)
0927 05:12:57.681098 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:57.681141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:57.682750 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:57.683418 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:57.683426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:57.683896 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:57.684415 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:57.684431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.685937 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.685941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:57.687494 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:57.687510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:57.688225 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:57.688234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:57.688486 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:57.689145 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:57.689165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:57.691220 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:57.694297 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:57.694307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:57.694320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:57.694703 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:57.695096 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:57.695102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:57.695106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:57.695281 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:57.695760 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:57.695767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:57.696061 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 05:12:57.696309 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 05:12:57.696775 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":328,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:12:57.699334 3159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45993
I20260927 05:12:57.699632 3159 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:57.700336 3159 heartbeater.cc:511] Master 127.0.97.62:45993 requested a full tablet report, sending...
I20260927 05:12:57.701759 2836 ts_manager.cc:194] Registered new tserver with Master: 487290dad2b748249268c58be6234fe3 (127.0.97.2:35599)
I20260927 05:12:57.702528 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:12:57.720444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:58.038749 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:58.039157 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:58.039341 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:58.046660 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:58.046753 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:58.046952 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:12:58.056401 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:58.057720 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:58.059085 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:58.071584 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:58.071589 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:58.071645 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:58.072521 3164 server_base.cc:1034] running on GCE node
I20260927 05:12:58.073083 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:58.073928 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:58.075093 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790485978075055 us; error 52 us; skew 500 ppm
I20260927 05:12:58.075387 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:58 dist-test-slave-1t6p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:58.078382 3164 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:12:58.079717 3164 webserver.cc:460] Webserver started at http://127.0.97.3:34669/ using document root <none> and password file <none>
I20260927 05:12:58.080370 3164 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:58.080509 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:58.080807 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:58.083528 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "9f764aa57f4d4003937784ae510d1f8e"
format_stamp: "Formatted at 2026-09-27 05:12:58 on dist-test-slave-1t6p"
server_key: "c637d4a3185501f4962d99c8751d44c8"
server_key_iv: "cfc2e0456faa1ca3278a3695bd094472"
server_key_version: "encryptionkey@0"
I20260927 05:12:58.084342 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9f764aa57f4d4003937784ae510d1f8e"
format_stamp: "Formatted at 2026-09-27 05:12:58 on dist-test-slave-1t6p"
server_key: "c637d4a3185501f4962d99c8751d44c8"
server_key_iv: "cfc2e0456faa1ca3278a3695bd094472"
server_key_version: "encryptionkey@0"
I20260927 05:12:58.089180 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:12:58.092623 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:58.093950 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:12:58.094121 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "9f764aa57f4d4003937784ae510d1f8e"
format_stamp: "Formatted at 2026-09-27 05:12:58 on dist-test-slave-1t6p"
server_key: "c637d4a3185501f4962d99c8751d44c8"
server_key_iv: "cfc2e0456faa1ca3278a3695bd094472"
server_key_version: "encryptionkey@0"
I20260927 05:12:58.094322 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:12:58.121608 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:12:58.125662 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:12:58.125963 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:12:58.126816 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:12:58.128252 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:12:58.128350 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:58.128492 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:12:58.128562 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:12:58.159245 3164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45949
I20260927 05:12:58.159324 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45949 every 8 connection(s)
I20260927 05:12:58.160437 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:12:58.165571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3164
I20260927 05:12:58.165757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:12:58.166313 388 external_mini_cluster.cc:1428] Setting key ec1dfe89327f2bdebc07b3e25f376ee2
Sep 27 05:12:58 dist-test-slave-1t6p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:12:58.178851 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:58.162777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44423 (local address 127.0.97.62:45993)
0927 05:12:58.162980 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:58.162986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:58.164169 (+ 1183us) server_negotiation.cc:407] Connection header received
0927 05:12:58.165986 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:58.165992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:58.166075 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:58.166191 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:58.167864 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:58.168892 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:58.170344 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:12:58.170735 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:12:58.170829 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:58.173959 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:12:58.173964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:12:58.173968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:12:58.174000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:12:58.176347 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:58.177106 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:58.177110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:58.177112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:58.177220 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:12:58.177567 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:12:58.177570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:12:58.177572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:12:58.177802 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:12:58.177918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:12:58.178450 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 05:12:58.178619 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:12:58.179804 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:12:58.163367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45993 (local address 127.0.97.3:44423)
0927 05:12:58.164011 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 05:12:58.164056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:12:58.165664 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:12:58.166419 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:12:58.166431 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:12:58.167067 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:12:58.167708 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:58.167725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:58.169039 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:58.169044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:58.170225 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:12:58.170235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:12:58.170917 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:12:58.170921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:12:58.171157 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:12:58.171800 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:12:58.171821 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:12:58.173772 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:12:58.176463 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:58.176469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:58.176480 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:58.177007 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:58.177304 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:12:58.177308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:12:58.177310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:12:58.177475 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:12:58.177898 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:12:58.177905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:12:58.178163 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 05:12:58.178358 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 05:12:58.178697 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":255,"threads_started":1}
I20260927 05:12:58.181200 3292 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45993
I20260927 05:12:58.181484 3292 heartbeater.cc:464] Registering TS with master...
I20260927 05:12:58.182127 3292 heartbeater.cc:511] Master 127.0.97.62:45993 requested a full tablet report, sending...
I20260927 05:12:58.183413 2836 ts_manager.cc:194] Registered new tserver with Master: 9f764aa57f4d4003937784ae510d1f8e (127.0.97.3:45949)
I20260927 05:12:58.184110 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44423
I20260927 05:12:58.193385 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:12:58.223825 3027 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:12:58.221465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.0.97.62:45993)
0927 05:12:58.221696 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:12:58.221703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:12:58.222002 (+ 299us) server_negotiation.cc:407] Connection header received
0927 05:12:58.222145 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:12:58.222149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:12:58.222212 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:12:58.222320 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:12:58.223564 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37144: BlockingRecv error: recv got EOF from 127.0.0.1:37144 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:12:58.226843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2897
W20260927 05:12:58.229444 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2897: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35497: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:12:58.229624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2897
I20260927 05:12:58.244686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3031
W20260927 05:12:58.247159 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3031: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35599: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:12:58.247313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3031
I20260927 05:12:58.262147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3164
W20260927 05:12:58.264993 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3164: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45949: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:12:58.265151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3164
I20260927 05:12:58.279937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 2804
I20260927 05:12:58.281004 2866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:12:58.380944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 2804
2026-09-27T05:12:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3413 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:12:58 dist-test-slave-1t6p krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:12:58 dist-test-slave-1t6p krb5kdc[3309](info): set up 2 sockets
Sep 27 05:12:58 dist-test-slave-1t6p krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260927 05:12:59.463169 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:12:59 dist-test-slave-1t6p krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:12:59.636562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:12:59.944558 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:12:59.944977 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:12:59.945148 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:12:59.953460 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:12:59.953547 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:12:59.953655 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:12:59.953756 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:12:59.953825 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:12:59.963209 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:12:59.964406 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:12:59.965626 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:12:59.976862 3327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:59.976776 3328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:12:59.976827 3330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:12:59.977876 3322 server_base.cc:1034] running on GCE node
I20260927 05:12:59.978642 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:12:59.979940 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:12:59.981128 3322 hybrid_clock.cc:648] HybridClock initialized: now 1790485979981089 us; error 31 us; skew 500 ppm
I20260927 05:12:59.981460 3322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:12:59 dist-test-slave-1t6p krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:12:59.988349 3322 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:12:59.989763 3322 webserver.cc:460] Webserver started at http://127.0.97.62:42705/ using document root <none> and password file <none>
I20260927 05:12:59.990423 3322 fs_manager.cc:359] Metadata directory not provided
I20260927 05:12:59.990564 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:12:59.990856 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:12:59.993683 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "5aaba840d908475ca56d9e7220f0a40b"
format_stamp: "Formatted at 2026-09-27 05:12:59 on dist-test-slave-1t6p"
server_key: "88e9efdd437078f56acd336168862e38"
server_key_iv: "b5ceb1c972cb347e4e4bd05e77160e5d"
server_key_version: "encryptionkey@0"
I20260927 05:12:59.994458 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "5aaba840d908475ca56d9e7220f0a40b"
format_stamp: "Formatted at 2026-09-27 05:12:59 on dist-test-slave-1t6p"
server_key: "88e9efdd437078f56acd336168862e38"
server_key_iv: "b5ceb1c972cb347e4e4bd05e77160e5d"
server_key_version: "encryptionkey@0"
I20260927 05:12:59.999328 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260927 05:13:00.002470 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:00.003713 3322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:00.003894 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "5aaba840d908475ca56d9e7220f0a40b"
format_stamp: "Formatted at 2026-09-27 05:12:59 on dist-test-slave-1t6p"
server_key: "88e9efdd437078f56acd336168862e38"
server_key_iv: "b5ceb1c972cb347e4e4bd05e77160e5d"
server_key_version: "encryptionkey@0"
I20260927 05:13:00.004074 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:00.034008 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:00.037762 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:00.038044 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:00.055914 3322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35509
I20260927 05:13:00.055948 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35509 every 8 connection(s)
I20260927 05:13:00.057238 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:00.060459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3322
I20260927 05:13:00.060674 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:00.060971 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:00.061332 388 external_mini_cluster.cc:1428] Setting key a2c3c5f7695a52df40e7194b42ac0412
I20260927 05:13:00.073352 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b: Bootstrap starting.
Sep 27 05:13:00 dist-test-slave-1t6p krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:00.076944 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:00.078097 3389 log.cc:824] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:00.080777 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b: No bootstrap required, opened a new log
I20260927 05:13:00.082590 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:00.063122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.97.62:35509)
0927 05:13:00.063906 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:00.063918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:00.063956 (+ 38us) server_negotiation.cc:407] Connection header received
0927 05:13:00.066111 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:00.066132 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:00.066508 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:00.066910 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:00.068347 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:00.069575 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:00.070755 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:00.071518 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:00.071713 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:00.075772 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:00.075779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:00.075792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:00.075825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:00.078819 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:00.079546 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:00.079552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:00.079558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:00.079704 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:00.080071 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:00.080080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:00.080082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:00.080554 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:00.080867 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:00.081266 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 05:13:00.081659 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":348,"threads_started":1}
I20260927 05:13:00.086248 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:13:00.088531 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } }
I20260927 05:13:00.088877 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:00.088982 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaba840d908475ca56d9e7220f0a40b, State: Initialized, Role: FOLLOWER
I20260927 05:13:00.089498 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } }
I20260927 05:13:00.089720 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:00.089861 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:00.090029 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:00.092130 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } }
I20260927 05:13:00.092594 3389 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aaba840d908475ca56d9e7220f0a40b; no voters:
I20260927 05:13:00.093251 3389 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:00.093361 3395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:00.095142 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:00.095142 3395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [term 1 LEADER]: Becoming Leader. State: Replica: 5aaba840d908475ca56d9e7220f0a40b, State: Running, Role: LEADER
I20260927 05:13:00.095762 3395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } }
I20260927 05:13:00.098547 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaba840d908475ca56d9e7220f0a40b. Latest consensus state: current_term: 1 leader_uuid: "5aaba840d908475ca56d9e7220f0a40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } } }
I20260927 05:13:00.098910 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:00.099364 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aaba840d908475ca56d9e7220f0a40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaba840d908475ca56d9e7220f0a40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35509 } } }
I20260927 05:13:00.099658 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:00.107719 3406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:00.112704 3406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:00.133962 3406 catalog_manager.cc:1347] Generated new cluster ID: 63742cc702e24b95b52517048efe9fe7
I20260927 05:13:00.134146 3406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:00.144881 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:00.136493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.97.62:35509)
0927 05:13:00.136640 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:00.136646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:00.137170 (+ 524us) server_negotiation.cc:407] Connection header received
0927 05:13:00.137326 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:00.137332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:00.137414 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:00.137529 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:00.138566 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:00.139696 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:00.141415 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:00.142077 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:00.142209 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:00.143349 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:00.143356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:00.143361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:00.144115 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:00.144191 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:00.144218 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:00.144337 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:00.144429 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 05:13:00.144514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:13:00.157112 3406 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:00.158804 3406 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:00.167665 3406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aaba840d908475ca56d9e7220f0a40b: Generated new TSK 0
I20260927 05:13:00.168406 3406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:13:00.180488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3322
I20260927 05:13:00.182384 3384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:00.270233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3322
2026-09-27T05:13:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1886 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:00 dist-test-slave-1t6p krb5kdc[3424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:00 dist-test-slave-1t6p krb5kdc[3424](info): set up 2 sockets
Sep 27 05:13:00 dist-test-slave-1t6p krb5kdc[3424](info): commencing operation
krb5kdc: starting...
W20260927 05:13:01.339200 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:01 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:01.518105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41349
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:01.827178 3437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:01.827484 3437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:01.827648 3437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:01.834792 3437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:01.834926 3437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:01.835041 3437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:01.835150 3437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:01.835215 3437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:01.844463 3437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:01.845685 3437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:01.846982 3437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:01.858348 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:01.858493 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:01.858502 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:01.859350 3437 server_base.cc:1034] running on GCE node
I20260927 05:13:01.860052 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:01.861335 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:01.862553 3437 hybrid_clock.cc:648] HybridClock initialized: now 1790485981862514 us; error 46 us; skew 500 ppm
I20260927 05:13:01.862881 3437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:01 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:01.865768 3437 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:01.867147 3437 webserver.cc:460] Webserver started at http://127.0.97.62:43253/ using document root <none> and password file <none>
I20260927 05:13:01.867789 3437 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:01.867924 3437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:01.868207 3437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:01.870898 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "c784db669b604b77aa81d581faa9d475"
format_stamp: "Formatted at 2026-09-27 05:13:01 on dist-test-slave-1t6p"
server_key: "51151e36f97aa504a2a4bbe0fb5227de"
server_key_iv: "ff321166a4a960fd8c9099f172489b05"
server_key_version: "encryptionkey@0"
I20260927 05:13:01.871678 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "c784db669b604b77aa81d581faa9d475"
format_stamp: "Formatted at 2026-09-27 05:13:01 on dist-test-slave-1t6p"
server_key: "51151e36f97aa504a2a4bbe0fb5227de"
server_key_iv: "ff321166a4a960fd8c9099f172489b05"
server_key_version: "encryptionkey@0"
I20260927 05:13:01.876499 3437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:01.879614 3452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:01.880844 3437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:13:01.881019 3437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "c784db669b604b77aa81d581faa9d475"
format_stamp: "Formatted at 2026-09-27 05:13:01 on dist-test-slave-1t6p"
server_key: "51151e36f97aa504a2a4bbe0fb5227de"
server_key_iv: "ff321166a4a960fd8c9099f172489b05"
server_key_version: "encryptionkey@0"
I20260927 05:13:01.881202 3437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:01.897035 3437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:01.901021 3437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:01.901292 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:01.919179 3437 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41349
I20260927 05:13:01.919216 3503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41349 every 8 connection(s)
I20260927 05:13:01.920472 3437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:01.923396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3437
I20260927 05:13:01.923615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:01.924170 3504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:01.924238 388 external_mini_cluster.cc:1428] Setting key 7b3f341cd3508f2e888e91cad1780df4
I20260927 05:13:01.937529 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475: Bootstrap starting.
Sep 27 05:13:01 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:01.940974 3504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:01.941998 3504 log.cc:824] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:01.944667 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475: No bootstrap required, opened a new log
I20260927 05:13:01.947332 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:01.926334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.0.97.62:41349)
0927 05:13:01.927297 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:01.927312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:01.927912 (+ 600us) server_negotiation.cc:407] Connection header received
0927 05:13:01.930084 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:01.930104 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:01.930473 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:01.930785 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:01.932469 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:01.933663 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:01.935421 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:01.936042 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:01.936161 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:01.940542 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:01.940550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:01.940564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:01.940600 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:01.943566 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:01.944181 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:01.944187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:01.944192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:01.944346 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:01.944707 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:01.944716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:01.944718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:01.945227 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:01.945536 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:01.945922 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 05:13:01.946334 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":407,"threads_started":1}
I20260927 05:13:01.953298 3504 raft_consensus.cc:348] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } }
I20260927 05:13:01.953773 3504 raft_consensus.cc:374] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:01.953905 3504 raft_consensus.cc:729] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c784db669b604b77aa81d581faa9d475, State: Initialized, Role: FOLLOWER
I20260927 05:13:01.954541 3504 consensus_queue.cc:260] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } }
I20260927 05:13:01.954844 3504 raft_consensus.cc:388] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:01.954994 3504 raft_consensus.cc:482] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:01.955193 3504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:01.957434 3504 raft_consensus.cc:504] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } }
I20260927 05:13:01.957893 3504 leader_election.cc:302] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c784db669b604b77aa81d581faa9d475; no voters:
I20260927 05:13:01.958359 3504 leader_election.cc:288] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:01.958527 3509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:01.959604 3509 raft_consensus.cc:686] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [term 1 LEADER]: Becoming Leader. State: Replica: c784db669b604b77aa81d581faa9d475, State: Running, Role: LEADER
I20260927 05:13:01.960139 3509 consensus_queue.cc:237] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } }
I20260927 05:13:01.960477 3504 sys_catalog.cc:564] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:01.962680 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c784db669b604b77aa81d581faa9d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } } }
I20260927 05:13:01.962833 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c784db669b604b77aa81d581faa9d475. Latest consensus state: current_term: 1 leader_uuid: "c784db669b604b77aa81d581faa9d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c784db669b604b77aa81d581faa9d475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41349 } } }
I20260927 05:13:01.963128 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:01.963176 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:01.964597 3517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:01.969338 3517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:01.978163 3517 catalog_manager.cc:1347] Generated new cluster ID: 29bafec7f0594657880fea674d0af813
I20260927 05:13:01.978282 3517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:01.995985 3517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:01.997256 3517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:02.005808 3517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c784db669b604b77aa81d581faa9d475: Generated new TSK 0
I20260927 05:13:02.006763 3517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:13:02.062634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:02.372648 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:02.372893 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:02.372987 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:02.379850 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:02.379968 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:02.380151 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:02.389346 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:02.390494 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:02.391767 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:02.404559 3535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:02.404619 3536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:02.404652 3538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:02.404942 3530 server_base.cc:1034] running on GCE node
I20260927 05:13:02.405584 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:02.406425 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:02.407639 3530 hybrid_clock.cc:648] HybridClock initialized: now 1790485982407606 us; error 44 us; skew 500 ppm
I20260927 05:13:02.407931 3530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:02 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:02.410807 3530 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:13:02.412096 3530 webserver.cc:460] Webserver started at http://127.0.97.1:44957/ using document root <none> and password file <none>
I20260927 05:13:02.412719 3530 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:02.412858 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:02.413134 3530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:02.415850 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "89cbba2a1bf842f3a200ac7d788535ce"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "3324568d09c21315789eb368e0fe8615"
server_key_iv: "6261e45c8250671eb06a47a80afd16f2"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.416584 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "89cbba2a1bf842f3a200ac7d788535ce"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "3324568d09c21315789eb368e0fe8615"
server_key_iv: "6261e45c8250671eb06a47a80afd16f2"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.421173 3530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 05:13:02.424296 3545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.425416 3530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:02.425590 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "89cbba2a1bf842f3a200ac7d788535ce"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "3324568d09c21315789eb368e0fe8615"
server_key_iv: "6261e45c8250671eb06a47a80afd16f2"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.425789 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:02.453372 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:02.457370 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:02.457656 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:02.458477 3530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:02.459719 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:02.459815 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.459959 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:02.460031 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.492120 3530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42989
I20260927 05:13:02.492163 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42989 every 8 connection(s)
I20260927 05:13:02.493338 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:02.499680 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3530
I20260927 05:13:02.499894 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:02.500509 388 external_mini_cluster.cc:1428] Setting key 190e7ca723e8393f52b49942cad4ac3f
Sep 27 05:13:02 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:02.514413 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:02.495955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39155 (local address 127.0.97.62:41349)
0927 05:13:02.496359 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:02.496364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:02.497316 (+ 952us) server_negotiation.cc:407] Connection header received
0927 05:13:02.499038 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:02.499044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:02.499138 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:02.499256 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:02.500832 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.501608 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.503153 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.503555 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.503685 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:02.507546 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:02.507551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:02.507554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:02.507585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:02.511153 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:02.511866 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:02.511874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:02.511880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:02.512042 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:02.512508 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:02.512516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:02.512523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:02.512892 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:02.513056 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:02.513672 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 05:13:02.513944 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260927 05:13:02.514981 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:02.496526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41349 (local address 127.0.97.1:39155)
0927 05:13:02.497162 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:02.497197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:02.498748 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:02.499484 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:02.499492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:02.500092 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:02.500677 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:02.500694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.501748 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.501754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:02.502971 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:02.502982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.503767 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.503775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:02.504011 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:02.504677 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:02.504698 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:02.507331 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:02.511317 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:02.511324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:02.511336 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:02.511738 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:02.512193 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:02.512197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:02.512200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:02.512386 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:02.513021 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:02.513026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:02.513326 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 05:13:02.513542 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 05:13:02.513908 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":254,"threads_started":1}
I20260927 05:13:02.516691 3658 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41349
I20260927 05:13:02.517005 3658 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:02.517689 3658 heartbeater.cc:511] Master 127.0.97.62:41349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:13:02.519793 3469 ts_manager.cc:194] Registered new tserver with Master: 89cbba2a1bf842f3a200ac7d788535ce (127.0.97.1:42989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:13:02.521317 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:13:02.541800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:02.849979 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:02.850227 3664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:02.850376 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:02.857318 3664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:02.857411 3664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:02.857595 3664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:02.866993 3664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:02.868253 3664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:02.869504 3664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:02.882210 3669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:02.882187 3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:02.882196 3670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:02.882907 3664 server_base.cc:1034] running on GCE node
I20260927 05:13:02.883487 3664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:02.884349 3664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:02.885532 3664 hybrid_clock.cc:648] HybridClock initialized: now 1790485982885496 us; error 27 us; skew 500 ppm
I20260927 05:13:02.885862 3664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:02 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:02.888859 3664 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:13:02.890151 3664 webserver.cc:460] Webserver started at http://127.0.97.2:38265/ using document root <none> and password file <none>
I20260927 05:13:02.890793 3664 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:02.890937 3664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:02.891261 3664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:02.893985 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "3a651c4b919b4034909ec60d5930fe0d"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "493fff609e307562e64c45bc6b5ff3ac"
server_key_iv: "885373ad3d97ef079114e2263c6fb308"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.894730 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a651c4b919b4034909ec60d5930fe0d"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "493fff609e307562e64c45bc6b5ff3ac"
server_key_iv: "885373ad3d97ef079114e2263c6fb308"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.900033 3664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:13:02.903162 3679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.904357 3664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 05:13:02.904520 3664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "3a651c4b919b4034909ec60d5930fe0d"
format_stamp: "Formatted at 2026-09-27 05:13:02 on dist-test-slave-1t6p"
server_key: "493fff609e307562e64c45bc6b5ff3ac"
server_key_iv: "885373ad3d97ef079114e2263c6fb308"
server_key_version: "encryptionkey@0"
I20260927 05:13:02.904708 3664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:02.929443 3664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:02.933446 3664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:02.933749 3664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:02.934644 3664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:02.935940 3664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:02.936038 3664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.936192 3664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:02.936264 3664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:02.969309 3664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41991
I20260927 05:13:02.969360 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41991 every 8 connection(s)
I20260927 05:13:02.970554 3664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:02.977821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3664
I20260927 05:13:02.978025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:02.978434 388 external_mini_cluster.cc:1428] Setting key 6315d54ab41a5f48cc666f964175d986
Sep 27 05:13:02 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:02.990293 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:02.973164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49065 (local address 127.0.97.62:41349)
0927 05:13:02.973335 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:02.973339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:02.974461 (+ 1122us) server_negotiation.cc:407] Connection header received
0927 05:13:02.976258 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:02.976263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:02.976351 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:02.976461 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:02.977903 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.978759 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.980577 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.980979 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.981108 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:02.984757 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:02.984762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:02.984766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:02.984795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:02.987418 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:02.988105 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:02.988115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:02.988117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:02.988243 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:02.988654 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:02.988664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:02.988666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:02.988905 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:02.989056 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:02.989712 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 05:13:02.989935 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:02.991307 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:02.973571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41349 (local address 127.0.97.2:49065)
0927 05:13:02.974281 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:02.974323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:02.976000 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:02.976689 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:02.976697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:02.977187 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:02.977734 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:02.977750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.978905 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.978910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:02.980431 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:02.980443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:02.981191 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:02.981198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:02.981419 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:02.982092 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:02.982114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:02.984559 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:02.987550 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:02.987559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:02.987576 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:02.987982 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:02.988348 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:02.988355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:02.988361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:02.988538 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:02.989033 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:02.989041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:02.989364 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 05:13:02.989617 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 05:13:02.990142 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260927 05:13:02.992729 3792 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41349
I20260927 05:13:02.993074 3792 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:02.993958 3792 heartbeater.cc:511] Master 127.0.97.62:41349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:13:02.995627 3469 ts_manager.cc:194] Registered new tserver with Master: 3a651c4b919b4034909ec60d5930fe0d (127.0.97.2:41991)
I20260927 05:13:02.996399 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:13:03.020388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:03.331480 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:03.331722 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:03.331898 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:03.338770 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:03.338905 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:03.339107 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:03.348080 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:03.349239 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:03.350481 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:03.362789 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:03.362864 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:03.362885 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:03.368342 3797 server_base.cc:1034] running on GCE node
I20260927 05:13:03.368860 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:03.369654 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:03.370843 3797 hybrid_clock.cc:648] HybridClock initialized: now 1790485983370800 us; error 33 us; skew 500 ppm
I20260927 05:13:03.371192 3797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:03.374634 3797 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:13:03.375981 3797 webserver.cc:460] Webserver started at http://127.0.97.3:44043/ using document root <none> and password file <none>
I20260927 05:13:03.376616 3797 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:03.376706 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:03.376938 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:03.379657 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "060d0462fafb439bb7293a2adfd2ea51"
format_stamp: "Formatted at 2026-09-27 05:13:03 on dist-test-slave-1t6p"
server_key: "bc0c8de0d35bfa4944362421adbfcaa7"
server_key_iv: "9698f4e146b382a399c193121758d1f9"
server_key_version: "encryptionkey@0"
I20260927 05:13:03.380334 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "060d0462fafb439bb7293a2adfd2ea51"
format_stamp: "Formatted at 2026-09-27 05:13:03 on dist-test-slave-1t6p"
server_key: "bc0c8de0d35bfa4944362421adbfcaa7"
server_key_iv: "9698f4e146b382a399c193121758d1f9"
server_key_version: "encryptionkey@0"
I20260927 05:13:03.385072 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:03.388278 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:03.389526 3797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:03.389683 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "060d0462fafb439bb7293a2adfd2ea51"
format_stamp: "Formatted at 2026-09-27 05:13:03 on dist-test-slave-1t6p"
server_key: "bc0c8de0d35bfa4944362421adbfcaa7"
server_key_iv: "9698f4e146b382a399c193121758d1f9"
server_key_version: "encryptionkey@0"
I20260927 05:13:03.389868 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:03.407202 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:03.411147 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:03.411382 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:03.412210 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:03.413401 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:03.413470 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:03.413549 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:03.413592 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:03.444449 3797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44805
I20260927 05:13:03.444492 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44805 every 8 connection(s)
I20260927 05:13:03.445642 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:03.446583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3797
I20260927 05:13:03.446797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:03.447363 388 external_mini_cluster.cc:1428] Setting key 9626a7caf971d0636e1c0e0b8795e08d
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:03.464682 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.448528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45803 (local address 127.0.97.62:41349)
0927 05:13:03.448731 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:03.448737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:03.449892 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 05:13:03.451807 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:03.451811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:03.451869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:03.451960 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:03.453414 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.454154 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.455708 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.456104 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.456196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.459248 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:03.459253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:03.459256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:03.459285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:03.462183 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.462896 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.462900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.462902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.463063 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.463401 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.463404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.463406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.463652 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:03.463753 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:03.464299 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 05:13:03.464445 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:13:03.465503 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.448885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41349 (local address 127.0.97.3:45803)
0927 05:13:03.449722 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:03.449769 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:03.451567 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:03.452153 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:03.452160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:03.452615 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:03.453271 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:03.453287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.454284 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.454287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:03.455593 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:03.455604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.456275 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.456279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:03.456455 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.457032 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:03.457056 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:03.459082 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:03.462301 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:03.462310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:03.462324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:03.462793 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:03.463141 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:03.463144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:03.463146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:03.463309 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:03.463749 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:03.463754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:03.464013 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 05:13:03.464212 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 05:13:03.464527 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":352,"threads_started":1}
I20260927 05:13:03.466743 3925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41349
I20260927 05:13:03.467166 3925 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:03.467810 3925 heartbeater.cc:511] Master 127.0.97.62:41349 requested a full tablet report, sending...
I20260927 05:13:03.469085 3469 ts_manager.cc:194] Registered new tserver with Master: 060d0462fafb439bb7293a2adfd2ea51 (127.0.97.3:44805)
I20260927 05:13:03.469931 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45803
I20260927 05:13:03.474507 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:03.518734 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.505807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.97.62:41349)
0927 05:13:03.505972 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:03.505976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:03.506465 (+ 489us) server_negotiation.cc:407] Connection header received
0927 05:13:03.506609 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:03.506613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:03.506672 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:03.506758 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:03.508193 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.508951 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.510160 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.510559 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.510646 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.514090 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:03.514094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:03.514098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:03.514127 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:03.516668 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.517275 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.517278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.517280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.517380 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.517683 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.517686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.517688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.517959 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:03.518128 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:03.518373 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 05:13:03.518524 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 05:13:03.519347 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56810
I20260927 05:13:03.521102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3530
I20260927 05:13:03.523983 3658 heartbeater.cc:503] Master 127.0.97.62:41349 was elected leader, sending a full tablet report...
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:03.535873 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.522029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.97.1:42989)
0927 05:13:03.522529 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:03.522536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:03.522560 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:13:03.522610 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:03.522616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:03.522816 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:03.522954 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:03.524071 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.525302 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.526461 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.526969 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.527116 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.530740 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:03.530749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:03.530764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:03.530791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:03.533358 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.533971 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.533977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.533979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.534091 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.534437 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.534443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.534446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.534745 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:03.535069 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:03.535289 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 05:13:03.535532 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
W20260927 05:13:03.536998 3653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43192
W20260927 05:13:03.538043 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:13:03.538201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3530
I20260927 05:13:03.553478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3664
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:13:03.568200 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.554216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.97.2:41991)
0927 05:13:03.554665 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:03.554672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:03.554694 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:13:03.554746 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:03.554752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:03.554943 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:03.555154 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:03.556258 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.557209 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.558407 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.558924 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.559074 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.562791 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:03.562801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:03.562813 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:03.562850 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:03.565379 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.566033 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.566039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.566041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.566215 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.566575 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.566581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.566584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.566893 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:03.567349 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:03.567622 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 05:13:03.567870 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":202,"threads_started":1}
W20260927 05:13:03.569307 3787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47106
W20260927 05:13:03.570272 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:13:03.570446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3664
I20260927 05:13:03.585263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3797
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485983, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:13:03.600184 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:03.586149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.97.3:44805)
0927 05:13:03.586704 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:03.586711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:03.586737 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:13:03.586793 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:03.586799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:03.587061 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:03.587196 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:03.588318 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.589309 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.590427 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:03.590950 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:03.591121 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:03.595125 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:03.595136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:03.595150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:03.595179 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:03.597886 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.598435 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.598441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.598444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.598552 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:03.598835 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:03.598838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:03.598840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:03.599200 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:03.599405 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:03.599613 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 05:13:03.599786 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":282,"threads_started":1}
W20260927 05:13:03.601366 3920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39838
W20260927 05:13:03.602368 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:13:03.602492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3797
I20260927 05:13:03.617643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3437
I20260927 05:13:03.618626 3499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:03.722919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3437
2026-09-27T05:13:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3455 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3942](info): set up 2 sockets
Sep 27 05:13:03 dist-test-slave-1t6p krb5kdc[3942](info): commencing operation
krb5kdc: starting...
W20260927 05:13:04.799479 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:04 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:04.978520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39223
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:05.298461 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:05.298727 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:05.298810 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:05.305691 3955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:05.305794 3955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:05.305913 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:05.306012 3955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:05.306072 3955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:05.315325 3955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:05.316502 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:05.317768 3955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:05.329209 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:05.329111 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:05.329151 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:05.329869 3955 server_base.cc:1034] running on GCE node
I20260927 05:13:05.330617 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:05.331868 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:05.333086 3955 hybrid_clock.cc:648] HybridClock initialized: now 1790485985333047 us; error 51 us; skew 500 ppm
I20260927 05:13:05.333412 3955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:05 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:05.336288 3955 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:05.337631 3955 webserver.cc:460] Webserver started at http://127.0.97.62:37379/ using document root <none> and password file <none>
I20260927 05:13:05.338253 3955 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:05.338407 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:05.338692 3955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:05.341411 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "462efedd5d0844ab82b47903f93d0699"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "506becd042f6bcdecdded0ecf961c59d"
server_key_iv: "5b10e0c45f9e9670af4e2c050701b13a"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.342114 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "462efedd5d0844ab82b47903f93d0699"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "506becd042f6bcdecdded0ecf961c59d"
server_key_iv: "5b10e0c45f9e9670af4e2c050701b13a"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.346617 3955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 05:13:05.349786 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:05.350934 3955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:05.351109 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "462efedd5d0844ab82b47903f93d0699"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "506becd042f6bcdecdded0ecf961c59d"
server_key_iv: "5b10e0c45f9e9670af4e2c050701b13a"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.351284 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:05.368651 3955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:05.372186 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:05.372447 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:05.389635 3955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39223
I20260927 05:13:05.389678 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39223 every 8 connection(s)
I20260927 05:13:05.390923 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:05.394378 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:05.394781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 3955
I20260927 05:13:05.394965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:05.395620 388 external_mini_cluster.cc:1428] Setting key 7a41c6fa68dc96f4e7f4fac6d34befb7
I20260927 05:13:05.407224 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699: Bootstrap starting.
Sep 27 05:13:05 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:05.410609 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:05.411921 4022 log.cc:824] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:05.414834 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699: No bootstrap required, opened a new log
I20260927 05:13:05.418111 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:05.397426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.97.62:39223)
0927 05:13:05.398140 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:05.398154 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:05.398190 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:13:05.400002 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:05.400021 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:05.400371 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:05.400686 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:05.402326 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.403562 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.405219 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.405854 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.405978 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:05.410320 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:05.410330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:05.410347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:05.410387 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:05.414000 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:05.414573 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:05.414580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:05.414588 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:05.414775 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:05.415204 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:05.415209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:05.415211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:05.415866 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:05.416145 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:05.416420 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 05:13:05.416844 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":341,"threads_started":1}
I20260927 05:13:05.423612 4022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } }
I20260927 05:13:05.423979 4022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:05.424096 4022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462efedd5d0844ab82b47903f93d0699, State: Initialized, Role: FOLLOWER
I20260927 05:13:05.424649 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } }
I20260927 05:13:05.424839 4022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:05.424935 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:05.425045 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:05.427250 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } }
I20260927 05:13:05.427683 4022 leader_election.cc:302] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 462efedd5d0844ab82b47903f93d0699; no voters:
I20260927 05:13:05.428155 4022 leader_election.cc:288] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:05.428287 4027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:05.428912 4027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [term 1 LEADER]: Becoming Leader. State: Replica: 462efedd5d0844ab82b47903f93d0699, State: Running, Role: LEADER
I20260927 05:13:05.429349 4027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } }
I20260927 05:13:05.430023 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:05.432053 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 462efedd5d0844ab82b47903f93d0699. Latest consensus state: current_term: 1 leader_uuid: "462efedd5d0844ab82b47903f93d0699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } } }
I20260927 05:13:05.432502 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:05.433306 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "462efedd5d0844ab82b47903f93d0699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462efedd5d0844ab82b47903f93d0699" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39223 } } }
I20260927 05:13:05.433647 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:05.439993 4032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:05.444299 4032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:05.451691 4032 catalog_manager.cc:1347] Generated new cluster ID: 01fc8a9bbcd049c8b70050c78e39fc23
I20260927 05:13:05.451804 4032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:05.466343 4032 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:05.467549 4032 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:05.475873 4032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 462efedd5d0844ab82b47903f93d0699: Generated new TSK 0
I20260927 05:13:05.476794 4032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:13:05.528591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:05.840070 4048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:05.840329 4048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:05.840498 4048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:05.847383 4048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:05.847474 4048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:05.847674 4048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:05.857055 4048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:05.858251 4048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:05.859674 4048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:05.872606 4053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:05.872696 4054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:05.872633 4056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:05.873606 4048 server_base.cc:1034] running on GCE node
I20260927 05:13:05.874171 4048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:05.875118 4048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:05.876359 4048 hybrid_clock.cc:648] HybridClock initialized: now 1790485985876309 us; error 59 us; skew 500 ppm
I20260927 05:13:05.876662 4048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:05 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:05.879720 4048 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:13:05.881114 4048 webserver.cc:460] Webserver started at http://127.0.97.1:34107/ using document root <none> and password file <none>
I20260927 05:13:05.881764 4048 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:05.881903 4048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:05.882212 4048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:05.885085 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "88cb717c9ab54d6187ad0c31c3fa2d70"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "4a6ab718d6be268b9a0fa438f7dcd615"
server_key_iv: "ff665ffce454f758de8f00c608530da6"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.885866 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "88cb717c9ab54d6187ad0c31c3fa2d70"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "4a6ab718d6be268b9a0fa438f7dcd615"
server_key_iv: "ff665ffce454f758de8f00c608530da6"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.890662 4048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 05:13:05.893895 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:05.895231 4048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:13:05.895394 4048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "88cb717c9ab54d6187ad0c31c3fa2d70"
format_stamp: "Formatted at 2026-09-27 05:13:05 on dist-test-slave-1t6p"
server_key: "4a6ab718d6be268b9a0fa438f7dcd615"
server_key_iv: "ff665ffce454f758de8f00c608530da6"
server_key_version: "encryptionkey@0"
I20260927 05:13:05.895586 4048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:05.905615 4048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:05.909384 4048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:05.909654 4048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:05.910542 4048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:05.911906 4048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:05.912012 4048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:05.912158 4048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:05.912228 4048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:05.945434 4048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35161
I20260927 05:13:05.945492 4175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35161 every 8 connection(s)
I20260927 05:13:05.946661 4048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:05.955780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4048
I20260927 05:13:05.955976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:05.956723 388 external_mini_cluster.cc:1428] Setting key 60409d32fc940ca1b0258e12ddf6fc3f
Sep 27 05:13:05 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:05.967634 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:05.949082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47973 (local address 127.0.97.62:39223)
0927 05:13:05.949481 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:05.949486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:05.950476 (+ 990us) server_negotiation.cc:407] Connection header received
0927 05:13:05.952262 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:05.952267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:05.952353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:05.952466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:05.953862 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.954650 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.956860 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.957277 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.957382 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:05.961467 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:05.961472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:05.961475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:05.961504 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:05.964302 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:05.965187 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:05.965195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:05.965201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:05.965338 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:05.965728 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:05.965736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:05.965743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:05.966128 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:05.966274 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:05.966870 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 05:13:05.967161 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":201,"threads_started":1}
I20260927 05:13:05.968199 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:05.949656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39223 (local address 127.0.97.1:47973)
0927 05:13:05.950328 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:05.950364 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:05.952009 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:05.952688 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:05.952696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:05.953173 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:05.953703 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:05.953718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.954786 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.954793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:05.956689 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:05.956713 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:05.957495 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:05.957510 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:05.957727 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:05.958469 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:05.958495 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:05.961263 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:05.964431 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:05.964442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:05.964459 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:05.965066 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:05.965438 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:05.965441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:05.965443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:05.965608 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:05.966248 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:05.966259 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:05.966526 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 05:13:05.966730 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 05:13:05.967129 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":277,"threads_started":1}
I20260927 05:13:05.970737 4176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39223
I20260927 05:13:05.971127 4176 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:05.971951 4176 heartbeater.cc:511] Master 127.0.97.62:39223 requested a full tablet report, sending...
I20260927 05:13:05.974161 3987 ts_manager.cc:194] Registered new tserver with Master: 88cb717c9ab54d6187ad0c31c3fa2d70 (127.0.97.1:35161)
I20260927 05:13:05.975876 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47973
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:13:05.998368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:06.301342 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:06.301604 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:06.301797 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:06.308646 4182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:06.308763 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:06.308940 4182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:06.317943 4182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:06.319085 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:06.320303 4182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:06.332881 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:06.332852 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:06.332952 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:06.333588 4182 server_base.cc:1034] running on GCE node
I20260927 05:13:06.334154 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:06.334970 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:06.336211 4182 hybrid_clock.cc:648] HybridClock initialized: now 1790485986336165 us; error 43 us; skew 500 ppm
I20260927 05:13:06.336494 4182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:06.339754 4182 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:13:06.340978 4182 webserver.cc:460] Webserver started at http://127.0.97.2:37633/ using document root <none> and password file <none>
I20260927 05:13:06.341542 4182 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:06.341634 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:06.341866 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:06.344564 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "36c43ee522a541d78c30c41e85751305"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "339879e937ffbbc71eb13479be484d38"
server_key_iv: "1c4251f4ab789f5cdfd70d0be1820dfd"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.345252 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "36c43ee522a541d78c30c41e85751305"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "339879e937ffbbc71eb13479be484d38"
server_key_iv: "1c4251f4ab789f5cdfd70d0be1820dfd"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.349875 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:13:06.352900 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.354084 4182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:06.354264 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "36c43ee522a541d78c30c41e85751305"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "339879e937ffbbc71eb13479be484d38"
server_key_iv: "1c4251f4ab789f5cdfd70d0be1820dfd"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.354475 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:06.380328 4182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:06.384374 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:06.384593 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:06.385376 4182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:06.386500 4182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:06.386567 4182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.386644 4182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:06.386687 4182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.419584 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32985
I20260927 05:13:06.419633 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32985 every 8 connection(s)
I20260927 05:13:06.420723 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:06.423377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4182
I20260927 05:13:06.423538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:06.424063 388 external_mini_cluster.cc:1428] Setting key 19b253c31dd591ed349b1e5394626712
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:06.441468 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.422959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54001 (local address 127.0.97.62:39223)
0927 05:13:06.423170 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:06.423178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:06.424512 (+ 1334us) server_negotiation.cc:407] Connection header received
0927 05:13:06.426323 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:06.426326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:06.426396 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:06.426473 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:06.428584 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.429284 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.431759 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.432462 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.432594 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.435774 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:06.435779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:06.435782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:06.435812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:06.438548 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.439361 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.439366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.439368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.439485 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.439891 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.439896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.439899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.440150 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:06.440308 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:06.440930 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 05:13:06.441145 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:13:06.442348 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.423476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39223 (local address 127.0.97.2:54001)
0927 05:13:06.424328 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:06.424378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:06.426047 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:06.426742 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:06.426755 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:06.427583 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:06.428412 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:06.428436 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.429430 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.429436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:06.431600 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:06.431622 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.432660 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.432665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:06.432842 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.433494 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:06.433513 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:06.435566 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:06.438681 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:06.438690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:06.438705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:06.439231 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:06.439591 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:06.439597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:06.439601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:06.439773 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:06.440274 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:06.440282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:06.440594 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 05:13:06.440841 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 05:13:06.441316 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:13:06.443750 4310 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39223
I20260927 05:13:06.444079 4310 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:06.444820 4310 heartbeater.cc:511] Master 127.0.97.62:39223 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:13:06.446309 3987 ts_manager.cc:194] Registered new tserver with Master: 36c43ee522a541d78c30c41e85751305 (127.0.97.2:32985)
I20260927 05:13:06.447175 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:13:06.467347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:06.771649 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:06.771893 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:06.771981 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:06.778726 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:06.778827 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:06.778949 4315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:06.787900 4315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:06.788959 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:06.790129 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:06.802529 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:06.802614 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:06.802650 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:06.803360 4315 server_base.cc:1034] running on GCE node
I20260927 05:13:06.803928 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:06.804800 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:06.805992 4315 hybrid_clock.cc:648] HybridClock initialized: now 1790485986805941 us; error 45 us; skew 500 ppm
I20260927 05:13:06.806275 4315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:06.809141 4315 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:13:06.810417 4315 webserver.cc:460] Webserver started at http://127.0.97.3:34165/ using document root <none> and password file <none>
I20260927 05:13:06.811132 4315 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:06.811276 4315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:06.811579 4315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:06.814270 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "e1987e5cd71244ce816d3e23d562fd85"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "fe0b88db978b162f0e88c5e130322b1b"
server_key_iv: "b3540f76edc302aaafe2f98bd61f538d"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.815059 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e1987e5cd71244ce816d3e23d562fd85"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "fe0b88db978b162f0e88c5e130322b1b"
server_key_iv: "b3540f76edc302aaafe2f98bd61f538d"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.819690 4315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:13:06.822868 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.824080 4315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 05:13:06.824240 4315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "e1987e5cd71244ce816d3e23d562fd85"
format_stamp: "Formatted at 2026-09-27 05:13:06 on dist-test-slave-1t6p"
server_key: "fe0b88db978b162f0e88c5e130322b1b"
server_key_iv: "b3540f76edc302aaafe2f98bd61f538d"
server_key_version: "encryptionkey@0"
I20260927 05:13:06.824433 4315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:06.860090 4315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:06.864305 4315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:06.864583 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:06.865552 4315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:06.866855 4315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:06.866966 4315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.867137 4315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:06.867213 4315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:06.900110 4315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34723
I20260927 05:13:06.900192 4442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34723 every 8 connection(s)
I20260927 05:13:06.901319 4315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:06.902706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4315
I20260927 05:13:06.902875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:06.903450 388 external_mini_cluster.cc:1428] Setting key d421a2f1bda13c0524a2efcb1a180131
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:06.921036 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.903951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35057 (local address 127.0.97.62:39223)
0927 05:13:06.904151 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:06.904158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:06.905704 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 05:13:06.907746 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:06.907751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:06.907830 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:06.907947 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:06.909796 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.910538 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.912194 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.912602 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.912733 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.915865 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:06.915871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:06.915876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:06.915917 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:06.918623 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.919298 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.919302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.919304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.919414 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.919724 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.919727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.919729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.919972 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:06.920073 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:06.920647 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 05:13:06.920797 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:13:06.921846 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.904533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39223 (local address 127.0.97.3:35057)
0927 05:13:06.905550 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:06.905604 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:06.907500 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:06.908218 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:06.908226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:06.908956 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:06.909641 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:06.909658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.910688 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.910693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:06.912051 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:06.912062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.912817 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.912827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:06.913063 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.913621 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:06.913646 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:06.915675 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:06.918717 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:06.918723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:06.918734 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:06.919194 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:06.919481 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:06.919484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:06.919486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:06.919632 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:06.920071 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:06.920076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:06.920343 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 05:13:06.920547 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 05:13:06.920861 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":516,"threads_started":1}
I20260927 05:13:06.923095 4443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39223
I20260927 05:13:06.923380 4443 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:06.924129 4443 heartbeater.cc:511] Master 127.0.97.62:39223 requested a full tablet report, sending...
I20260927 05:13:06.925436 3987 ts_manager.cc:194] Registered new tserver with Master: e1987e5cd71244ce816d3e23d562fd85 (127.0.97.3:34723)
I20260927 05:13:06.926198 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35057
I20260927 05:13:06.930364 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:06.977806 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.963164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.0.97.1:35161)
0927 05:13:06.963755 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:06.963762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:06.963787 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:06.963842 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:06.963848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:06.964045 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:06.964167 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:06.965456 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.966415 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.967635 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.968156 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.968249 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.972080 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:06.972097 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:06.972113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:06.972147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:06.975514 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.976066 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.976073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.976075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.976184 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.976482 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.976485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.976487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.976775 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:06.976989 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:06.977224 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 05:13:06.977398 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":319,"threads_started":1}
I20260927 05:13:06.978737 4176 heartbeater.cc:503] Master 127.0.97.62:39223 was elected leader, sending a full tablet report...
Sep 27 05:13:06 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:06.993856 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.982140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.0.97.62:39223)
0927 05:13:06.982342 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:06.982348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:06.982452 (+ 104us) server_negotiation.cc:407] Connection header received
0927 05:13:06.982579 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:06.982582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:06.982636 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:06.982710 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:06.983846 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.984516 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.985662 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.986057 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.986142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:06.989552 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:06.989556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:06.989559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:06.989586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:06.991800 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.992325 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.992328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.992330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.992432 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:06.992710 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:06.992713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:06.992715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:06.993000 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:06.993164 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:06.993408 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 05:13:06.993542 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 05:13:06.994405 3987 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55984
I20260927 05:13:06.995376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4048
I20260927 05:13:07.006814 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:06.996134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.97.1:35161)
0927 05:13:06.996350 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:06.996356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:06.996650 (+ 294us) server_negotiation.cc:407] Connection header received
0927 05:13:06.996780 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:06.996783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:06.996836 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:06.996907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:06.998025 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:06.998695 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:06.999927 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:07.000335 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:07.000422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:07.002293 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:07.002298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:07.002301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:07.002328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:07.004843 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.005341 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.005344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.005346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.005453 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.005733 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.005736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.005738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.005964 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:07.006119 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:07.006352 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 05:13:07.006485 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:13:07.007323 4171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:07.088224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4048
I20260927 05:13:07.103801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4182
Sep 27 05:13:07 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:13:07.118249 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:07.104695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.0.97.2:32985)
0927 05:13:07.105198 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:07.105205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:07.105230 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:07.105288 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:07.105294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:07.105500 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:07.105626 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:07.106768 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:07.107831 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:07.109011 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:07.109525 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:07.109621 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:07.113479 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:07.113489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:07.113502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:07.113533 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:07.116073 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.116611 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.116617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.116619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.116731 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.117037 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.117040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.117042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.117351 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:07.117557 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:07.117777 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 05:13:07.117976 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260927 05:13:07.119665 4305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:07.199257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4182
I20260927 05:13:07.214375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4315
Sep 27 05:13:07 dist-test-slave-1t6p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:13:07.230250 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:07.215336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.97.3:34723)
0927 05:13:07.215916 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:07.215923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:07.215948 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:07.216003 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:07.216009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:07.216213 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:07.216384 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:07.217642 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:07.218870 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:07.220213 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:07.220791 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:07.220932 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:07.225051 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:07.225062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:07.225074 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:07.225107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:07.227887 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.228488 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.228494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.228496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.228624 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:07.228999 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:07.229003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:07.229005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:07.229317 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:07.229569 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:07.229806 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 05:13:07.229987 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":293,"threads_started":1}
I20260927 05:13:07.231504 4438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:07.312201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4315
I20260927 05:13:07.327436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 3955
I20260927 05:13:07.328465 4017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:07.426487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 3955
2026-09-27T05:13:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3700 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:07 dist-test-slave-1t6p krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:07 dist-test-slave-1t6p krb5kdc[4462](info): set up 2 sockets
Sep 27 05:13:07 dist-test-slave-1t6p krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20260927 05:13:08.499497 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:08 dist-test-slave-1t6p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:08.678572 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40595
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:08.985877 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:08.986147 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:08.986320 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:08.993472 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:08.993567 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:08.993685 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:08.993780 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:08.993873 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:09.003418 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:09.004622 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:09.005913 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:09.016918 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:09.017060 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:09.017274 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:09.017904 4475 server_base.cc:1034] running on GCE node
I20260927 05:13:09.018589 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:09.019877 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:09.021097 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790485989021068 us; error 47 us; skew 500 ppm
I20260927 05:13:09.021452 4475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:09 dist-test-slave-1t6p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:09.024385 4475 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:09.025384 4475 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:09.025465 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:09.025691 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:09.028451 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "c5e3240f5115446fb25230915544f48b"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "13841dd24989c6a698316727377c2b0b"
server_key_iv: "ced3b8d5281fddc72af3d897d0ea8484"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.029143 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5e3240f5115446fb25230915544f48b"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "13841dd24989c6a698316727377c2b0b"
server_key_iv: "ced3b8d5281fddc72af3d897d0ea8484"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.033478 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:09.036501 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:09.037672 4475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:09.037799 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "c5e3240f5115446fb25230915544f48b"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "13841dd24989c6a698316727377c2b0b"
server_key_iv: "ced3b8d5281fddc72af3d897d0ea8484"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.037966 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:09.059996 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:09.063484 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:09.063675 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:09.080873 4475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40595
I20260927 05:13:09.080892 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40595 every 8 connection(s)
I20260927 05:13:09.081717 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:09.083654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4475
I20260927 05:13:09.083835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:09.084486 388 external_mini_cluster.cc:1428] Setting key 39ae37f863a3ec8cb21b4d0d1d560121
I20260927 05:13:09.085430 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:09.096297 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: Bootstrap starting.
Sep 27 05:13:09 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:09.099660 4541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:09.100654 4541 log.cc:824] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:09.103518 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: No bootstrap required, opened a new log
I20260927 05:13:09.104633 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:09.086068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.0.97.62:40595)
0927 05:13:09.086709 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:09.086719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:09.086759 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:13:09.088379 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:09.088398 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:09.088748 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:09.089120 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:09.090511 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.091866 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.093057 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.093710 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.093895 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:09.098002 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:09.098009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:09.098022 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:09.098058 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:09.101072 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:09.101717 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:09.101724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:09.101738 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:09.101886 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:09.102293 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:09.102296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:09.102298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:09.102736 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:09.102915 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:09.103423 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 05:13:09.103761 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":306,"threads_started":1}
I20260927 05:13:09.110639 4541 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260927 05:13:09.110970 4541 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:09.111109 4541 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3240f5115446fb25230915544f48b, State: Initialized, Role: FOLLOWER
I20260927 05:13:09.111711 4541 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260927 05:13:09.112020 4541 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:09.112197 4541 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:09.112381 4541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:09.114353 4541 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260927 05:13:09.114845 4541 leader_election.cc:302] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e3240f5115446fb25230915544f48b; no voters:
I20260927 05:13:09.115393 4541 leader_election.cc:288] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:09.115510 4546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:09.116195 4546 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [term 1 LEADER]: Becoming Leader. State: Replica: c5e3240f5115446fb25230915544f48b, State: Running, Role: LEADER
I20260927 05:13:09.116711 4546 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260927 05:13:09.117547 4541 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:09.119473 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e3240f5115446fb25230915544f48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } } }
I20260927 05:13:09.119930 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:09.119933 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e3240f5115446fb25230915544f48b. Latest consensus state: current_term: 1 leader_uuid: "c5e3240f5115446fb25230915544f48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3240f5115446fb25230915544f48b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } } }
I20260927 05:13:09.120235 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:09.121606 4555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:09.125810 4555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:09.132855 4555 catalog_manager.cc:1347] Generated new cluster ID: d45bbfe1a8de436c8e6d4003db8de156
I20260927 05:13:09.132949 4555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:09.158103 4555 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:09.159494 4555 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:09.168566 4555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: Generated new TSK 0
I20260927 05:13:09.169548 4555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:13:09.224548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:09.529690 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:09.529982 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:09.530151 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:09.537101 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:09.537245 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:09.537422 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:09.547045 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:09.548257 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:09.549487 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:09.561839 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:09.561767 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:09.561862 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:09.562547 4567 server_base.cc:1034] running on GCE node
I20260927 05:13:09.563095 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:09.563903 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:09.565126 4567 hybrid_clock.cc:648] HybridClock initialized: now 1790485989565095 us; error 46 us; skew 500 ppm
I20260927 05:13:09.565418 4567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:09 dist-test-slave-1t6p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:09.568377 4567 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:13:09.569437 4567 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:09.569566 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:09.569857 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:09.572532 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "dffcaabdd22e438289cdc45c1cf0ffdb"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "297f12e06b5be48ca463b57ac747d7c9"
server_key_iv: "79c384d2a96bb43cdbff58967289b896"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.573287 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dffcaabdd22e438289cdc45c1cf0ffdb"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "297f12e06b5be48ca463b57ac747d7c9"
server_key_iv: "79c384d2a96bb43cdbff58967289b896"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.578300 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:13:09.581449 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:09.582659 4567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:13:09.582828 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "dffcaabdd22e438289cdc45c1cf0ffdb"
format_stamp: "Formatted at 2026-09-27 05:13:09 on dist-test-slave-1t6p"
server_key: "297f12e06b5be48ca463b57ac747d7c9"
server_key_iv: "79c384d2a96bb43cdbff58967289b896"
server_key_version: "encryptionkey@0"
I20260927 05:13:09.583079 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:09.605602 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:09.609575 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:09.609835 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:09.610744 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:09.612042 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:09.612169 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:09.612290 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:09.612394 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:09.644116 4567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37533
I20260927 05:13:09.644193 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37533 every 8 connection(s)
I20260927 05:13:09.645008 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:09.649842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4567
I20260927 05:13:09.650028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:09.650826 388 external_mini_cluster.cc:1428] Setting key 035538ca4171cea68e499f50ed6dfde3
Sep 27 05:13:09 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:09.668848 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:09.647285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34381 (local address 127.0.97.62:40595)
0927 05:13:09.647727 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:09.647732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:09.648513 (+ 781us) server_negotiation.cc:407] Connection header received
0927 05:13:09.650810 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:09.650815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:09.650903 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:09.651063 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:09.652890 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.653589 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.655125 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.655501 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.655631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:09.659509 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:09.659517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:09.659525 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:09.659566 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:09.663254 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:09.665179 (+ 1925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:09.665184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:09.665188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:09.665364 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:09.665937 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:09.665942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:09.665945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:09.666299 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:09.666451 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:09.667522 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0927 05:13:09.668084 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":266,"threads_started":1}
I20260927 05:13:09.669240 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:09.647726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.1:34381)
0927 05:13:09.648361 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:09.648397 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:09.650517 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:09.651281 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:09.651289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:09.651977 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:09.652710 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:09.652736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.653730 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.653735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:09.654943 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:09.654953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:09.655710 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:09.655720 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:09.655951 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:09.656602 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:09.656623 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:09.659246 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:09.664234 (+ 4988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:09.664241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:09.664289 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:09.664996 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:09.665489 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:09.665494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:09.665498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:09.665758 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:09.666455 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:09.666465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:09.667031 (+ 566us) client_negotiation.cc:769] Sending connection context
0927 05:13:09.667364 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 05:13:09.667895 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":273,"threads_started":1}
I20260927 05:13:09.670930 4694 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260927 05:13:09.671278 4694 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:13:09.671998 4694 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:13:09.674091 4506 ts_manager.cc:194] Registered new tserver with Master: dffcaabdd22e438289cdc45c1cf0ffdb (127.0.97.1:37533)
I20260927 05:13:09.675486 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:13:09.694173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:10.002588 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:10.002903 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:10.003108 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:10.010129 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:10.010268 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:10.010449 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:10.019956 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:10.021209 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:10.022475 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:10.034854 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:10.034940 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:10.034917 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:10.035634 4700 server_base.cc:1034] running on GCE node
I20260927 05:13:10.036190 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:10.037055 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:10.038267 4700 hybrid_clock.cc:648] HybridClock initialized: now 1790485990038222 us; error 58 us; skew 500 ppm
I20260927 05:13:10.038578 4700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:10 dist-test-slave-1t6p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:10.041718 4700 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:13:10.042809 4700 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:10.042932 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:10.043285 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:10.046192 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "20ba234e29af4714b7764c0e73008318"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "c1eb2a3abfe447c9d5470a7b7c60f628"
server_key_iv: "d4f53305cfccfd68afdd89dd780695fc"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.046990 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "20ba234e29af4714b7764c0e73008318"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "c1eb2a3abfe447c9d5470a7b7c60f628"
server_key_iv: "d4f53305cfccfd68afdd89dd780695fc"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.052059 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:13:10.055274 4714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.056509 4700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:13:10.056685 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "20ba234e29af4714b7764c0e73008318"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "c1eb2a3abfe447c9d5470a7b7c60f628"
server_key_iv: "d4f53305cfccfd68afdd89dd780695fc"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.056886 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:10.076859 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:10.080941 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:10.081229 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:10.082181 4700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:10.083683 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:10.083765 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.083909 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:10.083986 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.117089 4700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38999
I20260927 05:13:10.117136 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38999 every 8 connection(s)
I20260927 05:13:10.117980 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:10.119951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4700
I20260927 05:13:10.120127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:10.120659 388 external_mini_cluster.cc:1428] Setting key ebc1001095ce6de3ff6d2051564adc02
Sep 27 05:13:10 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:13:10.139505 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.120665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54961 (local address 127.0.97.62:40595)
0927 05:13:10.120882 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.120888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.122249 (+ 1361us) server_negotiation.cc:407] Connection header received
0927 05:13:10.124110 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.124117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.124205 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.124317 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:10.125707 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.126876 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.128434 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.129171 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.129311 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.133047 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:10.133053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:10.133056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:10.133107 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:10.136626 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.137318 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.137323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.137326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.137462 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.137864 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.137868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.137872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.138122 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:10.138269 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.138918 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.139170 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:13:10.140506 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.121239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.2:54961)
0927 05:13:10.122080 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:10.122124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:10.123843 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:10.124497 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:10.124505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:10.124994 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:10.125531 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.125551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.127068 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.127080 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.128286 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.128297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.129364 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.129369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.129546 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.130224 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:10.130245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:10.132761 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:10.136766 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:10.136774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:10.136789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:10.137195 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:10.137573 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:10.137578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:10.137582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:10.137747 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:10.138244 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:10.138252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:10.138575 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 05:13:10.138823 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 05:13:10.139336 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:13:10.142104 4827 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260927 05:13:10.142410 4827 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:10.143136 4827 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
I20260927 05:13:10.144506 4506 ts_manager.cc:194] Registered new tserver with Master: 20ba234e29af4714b7764c0e73008318 (127.0.97.2:38999)
I20260927 05:13:10.145339 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:13:10.163964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:10.469827 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:10.470101 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:10.470269 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:10.477202 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:10.477339 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:10.477510 4832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:10.486852 4832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:10.488118 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:10.489351 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:10.501794 4837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:10.501739 4838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:10.501832 4840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:10.502516 4832 server_base.cc:1034] running on GCE node
I20260927 05:13:10.503201 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:10.504069 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:10.505288 4832 hybrid_clock.cc:648] HybridClock initialized: now 1790485990505244 us; error 65 us; skew 500 ppm
I20260927 05:13:10.505650 4832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:10 dist-test-slave-1t6p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:10.508756 4832 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:13:10.509848 4832 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:10.509972 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:10.510267 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:10.513659 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "d0e2019932d14b42b8299fed0b505f31"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "e69005ccb02887de8acb38f1978a019a"
server_key_iv: "99aaefd47e06a3e7119a7337956738bd"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.514866 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d0e2019932d14b42b8299fed0b505f31"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "e69005ccb02887de8acb38f1978a019a"
server_key_iv: "99aaefd47e06a3e7119a7337956738bd"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.521996 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 05:13:10.526695 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.528342 4832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:10.528525 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "d0e2019932d14b42b8299fed0b505f31"
format_stamp: "Formatted at 2026-09-27 05:13:10 on dist-test-slave-1t6p"
server_key: "e69005ccb02887de8acb38f1978a019a"
server_key_iv: "99aaefd47e06a3e7119a7337956738bd"
server_key_version: "encryptionkey@0"
I20260927 05:13:10.528801 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:10.573501 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:10.579217 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:10.579548 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:10.580785 4832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:10.582618 4832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:10.582732 4832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.582841 4832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:10.582927 4832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:10.636221 4832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34329
I20260927 05:13:10.636327 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34329 every 8 connection(s)
I20260927 05:13:10.637074 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:10.639573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 4832
I20260927 05:13:10.639758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:10.640381 388 external_mini_cluster.cc:1428] Setting key ccba2fe69a02adf4a0e112dbbda02bb0
Sep 27 05:13:10 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:10.662086 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.641514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33945 (local address 127.0.97.62:40595)
0927 05:13:10.645338 (+ 3824us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.645344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.645360 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:13:10.645424 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.645432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.645506 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.645602 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:10.647737 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.649198 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.650704 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.651579 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.651740 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.654945 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:10.654954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:10.654962 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:10.655031 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:10.659218 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.659933 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.659937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.659939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.660053 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.660461 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.660467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.660470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.660725 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:10.660879 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.661504 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.661715 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3722,"thread_start_us":4479,"threads_started":1}
I20260927 05:13:10.662961 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.640770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.3:33945)
0927 05:13:10.642083 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:10.642140 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:10.644732 (+ 2592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:10.646030 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:10.646041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:10.646781 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:10.647532 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.647559 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.649355 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.649359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.650566 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.650577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.651779 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.651784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.651972 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.652670 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:10.652690 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:10.654702 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:10.659343 (+ 4641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:10.659350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:10.659362 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:10.659759 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:10.660159 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:10.660167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:10.660171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:10.660344 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:10.660844 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:10.660851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:10.661164 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 05:13:10.661428 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 05:13:10.661898 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":4223,"threads_started":1}
I20260927 05:13:10.664355 4959 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260927 05:13:10.664611 4959 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:10.665148 4959 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
I20260927 05:13:10.666412 4506 ts_manager.cc:194] Registered new tserver with Master: d0e2019932d14b42b8299fed0b505f31 (127.0.97.3:34329)
I20260927 05:13:10.667230 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33945
I20260927 05:13:10.669126 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:13:10.678092 4694 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
I20260927 05:13:10.693172 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.673683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.0.97.62:40595)
0927 05:13:10.673864 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.673871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.678812 (+ 4941us) server_negotiation.cc:407] Connection header received
0927 05:13:10.678981 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.678989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.679098 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.679208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:10.680677 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.682016 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.683332 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.684173 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.684329 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.686588 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:10.686598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:10.686606 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:10.686650 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:10.690323 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.690896 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.690904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.690910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.691089 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:10.691414 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:10.691421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:10.691426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:10.691883 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:10.692156 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.692473 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.692724 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:10.697716 4506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:10.710813 4506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:10.747713 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.724079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38999 (local address 127.0.0.1:54476)
0927 05:13:10.734642 (+ 10563us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:10.734684 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:10.734879 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:10.737190 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:10.737201 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:10.737239 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:10.737683 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.737698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.740025 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.740033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.743732 (+ 3699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.743750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.746398 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.746406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.746745 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.746817 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 05:13:10.746968 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 05:13:10.747208 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10257,"thread_start_us":16730,"threads_started":1}
I20260927 05:13:10.748886 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.724642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.97.2:38999)
0927 05:13:10.736067 (+ 11425us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.736079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.736114 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:13:10.736370 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.736380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.736677 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.736870 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:10.737909 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.739844 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.743943 (+ 4099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.745866 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.746010 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.746270 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.748234 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.748448 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11209,"thread_start_us":11107,"threads_started":1}
I20260927 05:13:10.753266 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 7d749d134b724a929514363513ff854b (DEFAULT_TABLE table=test-table [id=215d9148177645a6b1b65c1b5d3d6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:10.754694 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d749d134b724a929514363513ff854b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:10.761430 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.724496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37533 (local address 127.0.0.1:56978)
0927 05:13:10.747951 (+ 23455us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:10.747968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:10.748059 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:10.752203 (+ 4144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:10.752208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:10.752233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:10.752380 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.752390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.754613 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.754621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.758041 (+ 3420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.758058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.760362 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.760370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.760598 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.760656 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 05:13:10.760742 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:13:10.760926 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23381,"thread_start_us":10526,"threads_started":1}
I20260927 05:13:10.762042 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.724187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34329 (local address 127.0.0.1:34230)
0927 05:13:10.741134 (+ 16947us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:10.741157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:10.741279 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:10.754817 (+ 13538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:10.754821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:10.754847 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:10.755419 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.755433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.757334 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.757340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.759383 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:10.759395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.761216 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.761223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:10.761409 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.761444 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:13:10.761512 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 05:13:10.761639 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16793,"mutex_wait_us":113,"thread_start_us":30390,"threads_started":1}
I20260927 05:13:10.763156 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.724768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.97.3:34329)
0927 05:13:10.750236 (+ 25468us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.750247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.750283 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:13:10.750381 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.750388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.750628 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.750810 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:10.755630 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.757147 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.759572 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.761000 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.762334 (+ 1334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.762550 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.762628 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.762772 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25298,"thread_start_us":27712,"threads_started":1}
I20260927 05:13:10.763674 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:10.725178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.0.97.1:37533)
0927 05:13:10.751478 (+ 26300us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:10.751486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:10.751511 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:10.751603 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:10.751609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:10.751806 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:10.751996 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:10.752928 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.754460 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.758235 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:10.760036 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:10.761635 (+ 1599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:10.761919 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:10.761996 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 05:13:10.762179 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26149,"thread_start_us":26016,"threads_started":1}
I20260927 05:13:10.766781 4894 tablet_service.cc:1467] Processing CreateTablet for tablet 7d749d134b724a929514363513ff854b (DEFAULT_TABLE table=test-table [id=215d9148177645a6b1b65c1b5d3d6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:10.766913 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 7d749d134b724a929514363513ff854b (DEFAULT_TABLE table=test-table [id=215d9148177645a6b1b65c1b5d3d6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:10.768211 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d749d134b724a929514363513ff854b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:10.768217 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d749d134b724a929514363513ff854b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:10.784075 4978 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Bootstrap starting.
I20260927 05:13:10.785022 4979 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Bootstrap starting.
I20260927 05:13:10.788079 4980 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Bootstrap starting.
I20260927 05:13:10.790330 4979 tablet_bootstrap.cc:654] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:10.790686 4978 tablet_bootstrap.cc:654] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:10.791770 4979 log.cc:824] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:10.792070 4978 log.cc:824] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:10.792501 4980 tablet_bootstrap.cc:654] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:10.793938 4980 log.cc:824] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:10.801695 4978 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: No bootstrap required, opened a new log
I20260927 05:13:10.802027 4978 ts_tablet_manager.cc:1397] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20260927 05:13:10.802979 4980 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: No bootstrap required, opened a new log
I20260927 05:13:10.803330 4980 ts_tablet_manager.cc:1397] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260927 05:13:10.805589 4979 tablet_bootstrap.cc:492] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: No bootstrap required, opened a new log
I20260927 05:13:10.805953 4979 ts_tablet_manager.cc:1397] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20260927 05:13:10.813512 4978 raft_consensus.cc:348] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.813978 4978 raft_consensus.cc:374] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:10.814103 4978 raft_consensus.cc:729] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ba234e29af4714b7764c0e73008318, State: Initialized, Role: FOLLOWER
I20260927 05:13:10.813863 4980 raft_consensus.cc:348] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.814257 4980 raft_consensus.cc:374] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:10.814352 4980 raft_consensus.cc:729] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e2019932d14b42b8299fed0b505f31, State: Initialized, Role: FOLLOWER
I20260927 05:13:10.814771 4978 consensus_queue.cc:260] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.814962 4980 consensus_queue.cc:260] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.815970 4827 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
I20260927 05:13:10.815932 4979 raft_consensus.cc:348] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.816299 4979 raft_consensus.cc:374] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:10.816387 4979 raft_consensus.cc:729] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffcaabdd22e438289cdc45c1cf0ffdb, State: Initialized, Role: FOLLOWER
I20260927 05:13:10.817019 4978 ts_tablet_manager.cc:1428] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:13:10.816900 4979 consensus_queue.cc:260] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:10.821969 4979 ts_tablet_manager.cc:1428] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260927 05:13:10.822865 4980 ts_tablet_manager.cc:1428] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260927 05:13:10.823235 4959 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
W20260927 05:13:10.869854 4828 tablet.cc:2369] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:10.897208 4695 tablet.cc:2369] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:10.897208 4960 tablet.cc:2369] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:11.061522 4984 raft_consensus.cc:482] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:11.061777 4984 raft_consensus.cc:504] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:11.063405 4984 leader_election.cc:288] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e2019932d14b42b8299fed0b505f31 (127.0.97.3:34329), dffcaabdd22e438289cdc45c1cf0ffdb (127.0.97.1:37533)
I20260927 05:13:11.070034 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.063765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45623 (local address 127.0.97.3:34329)
0927 05:13:11.064020 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.064025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.064923 (+ 898us) server_negotiation.cc:407] Connection header received
0927 05:13:11.065133 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.065138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.065204 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.065298 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:11.065854 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.066562 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.068427 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.069171 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.069319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.069432 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.069682 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.069801 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 05:13:11.070122 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.063898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51145 (local address 127.0.97.1:37533)
0927 05:13:11.064088 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.064094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.064989 (+ 895us) server_negotiation.cc:407] Connection header received
0927 05:13:11.065176 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.065181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.065248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.065340 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:11.065935 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.066641 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.068480 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.069234 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.069388 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.069503 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.069774 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.069888 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:13:11.070189 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.063921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34329 (local address 127.0.97.2:45623)
0927 05:13:11.064821 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:11.064836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:11.064998 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:11.065314 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:11.065318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:11.065340 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:11.065701 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:11.065710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.066703 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.066710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:11.068237 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:11.068274 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.069371 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.069377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:11.069529 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.069555 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:13:11.069659 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:13:11.069775 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":466,"threads_started":1}
I20260927 05:13:11.070207 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.064002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37533 (local address 127.0.97.2:51145)
0927 05:13:11.064882 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:11.064905 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:11.065051 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:11.065354 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:11.065358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:11.065378 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:11.065766 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:11.065774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.066770 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.066777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:11.068306 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:11.068317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.069467 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.069473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:11.069628 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.069650 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:13:11.069751 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:13:11.069864 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":485,"threads_started":1}
I20260927 05:13:11.071353 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d749d134b724a929514363513ff854b" candidate_uuid: "20ba234e29af4714b7764c0e73008318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e2019932d14b42b8299fed0b505f31" is_pre_election: true
I20260927 05:13:11.071364 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d749d134b724a929514363513ff854b" candidate_uuid: "20ba234e29af4714b7764c0e73008318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" is_pre_election: true
I20260927 05:13:11.071805 4914 raft_consensus.cc:2454] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20ba234e29af4714b7764c0e73008318 in term 0.
I20260927 05:13:11.071851 4649 raft_consensus.cc:2454] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20ba234e29af4714b7764c0e73008318 in term 0.
I20260927 05:13:11.072454 4717 leader_election.cc:302] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ba234e29af4714b7764c0e73008318, d0e2019932d14b42b8299fed0b505f31; no voters:
I20260927 05:13:11.072815 4984 raft_consensus.cc:2781] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:11.072970 4984 raft_consensus.cc:482] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:11.073082 4984 raft_consensus.cc:3037] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:11.075179 4984 raft_consensus.cc:504] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:11.075695 4984 leader_election.cc:288] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [CANDIDATE]: Term 1 election: Requested vote from peers d0e2019932d14b42b8299fed0b505f31 (127.0.97.3:34329), dffcaabdd22e438289cdc45c1cf0ffdb (127.0.97.1:37533)
I20260927 05:13:11.076110 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d749d134b724a929514363513ff854b" candidate_uuid: "20ba234e29af4714b7764c0e73008318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e2019932d14b42b8299fed0b505f31"
I20260927 05:13:11.076120 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d749d134b724a929514363513ff854b" candidate_uuid: "20ba234e29af4714b7764c0e73008318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb"
I20260927 05:13:11.076294 4914 raft_consensus.cc:3037] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:11.076310 4649 raft_consensus.cc:3037] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:11.078344 4649 raft_consensus.cc:2454] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20ba234e29af4714b7764c0e73008318 in term 1.
I20260927 05:13:11.078599 4914 raft_consensus.cc:2454] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20ba234e29af4714b7764c0e73008318 in term 1.
I20260927 05:13:11.078706 4718 leader_election.cc:302] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ba234e29af4714b7764c0e73008318, dffcaabdd22e438289cdc45c1cf0ffdb; no voters:
I20260927 05:13:11.078994 4984 raft_consensus.cc:2781] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:11.079442 4984 raft_consensus.cc:686] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 1 LEADER]: Becoming Leader. State: Replica: 20ba234e29af4714b7764c0e73008318, State: Running, Role: LEADER
I20260927 05:13:11.079887 4984 consensus_queue.cc:237] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } }
I20260927 05:13:11.083700 4505 catalog_manager.cc:5495] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20ba234e29af4714b7764c0e73008318 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "20ba234e29af4714b7764c0e73008318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ba234e29af4714b7764c0e73008318" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:11.113112 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.107969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.0.97.2:38999)
0927 05:13:11.108161 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.108166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.108312 (+ 146us) server_negotiation.cc:407] Connection header received
0927 05:13:11.108547 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.108552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.108623 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.108767 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:11.109115 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.109814 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.111122 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.111513 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.111645 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.112094 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:11.112565 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:11.112726 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.112782 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.112859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:13:11.121529 4914 raft_consensus.cc:1261] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 1 FOLLOWER]: Refusing update from remote peer 20ba234e29af4714b7764c0e73008318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:11.121557 4649 raft_consensus.cc:1261] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 1 FOLLOWER]: Refusing update from remote peer 20ba234e29af4714b7764c0e73008318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:11.122459 4989 consensus_queue.cc:1035] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e2019932d14b42b8299fed0b505f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:11.122818 4984 consensus_queue.cc:1035] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dffcaabdd22e438289cdc45c1cf0ffdb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:11.151261 4506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:11.151538 4506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:11.154635 4506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e3240f5115446fb25230915544f48b: Sending DeleteTablet for 3 replicas of tablet 7d749d134b724a929514363513ff854b
I20260927 05:13:11.155669 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d749d134b724a929514363513ff854b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260927 05:13:11.155687 4894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d749d134b724a929514363513ff854b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34230
I20260927 05:13:11.156507 5002 tablet_replica.cc:330] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: stopping tablet replica
I20260927 05:13:11.156841 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d749d134b724a929514363513ff854b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56978
I20260927 05:13:11.156932 5002 raft_consensus.cc:2229] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:11.157526 5002 raft_consensus.cc:2258] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:11.159303 5002 ts_tablet_manager.cc:1905] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:11.159332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4567
I20260927 05:13:11.167997 5002 ts_tablet_manager.cc:1918] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:11.168200 5002 log.cc:1196] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/7d749d134b724a929514363513ff854b
I20260927 05:13:11.168799 5002 ts_tablet_manager.cc:1939] T 7d749d134b724a929514363513ff854b P 20ba234e29af4714b7764c0e73008318: Deleting consensus metadata
I20260927 05:13:11.170670 4490 catalog_manager.cc:4841] TS 20ba234e29af4714b7764c0e73008318 (127.0.97.2:38999): tablet 7d749d134b724a929514363513ff854b (table test-table [id=215d9148177645a6b1b65c1b5d3d6827]) successfully deleted
I20260927 05:13:11.172070 5004 tablet_replica.cc:330] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: stopping tablet replica
I20260927 05:13:11.172547 5004 raft_consensus.cc:2229] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:11.173101 5004 raft_consensus.cc:2258] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:11.173765 5003 tablet_replica.cc:330] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: stopping tablet replica
I20260927 05:13:11.174083 5003 raft_consensus.cc:2229] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:11.174432 5003 raft_consensus.cc:2258] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:11.176064 5004 ts_tablet_manager.cc:1905] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:11.176182 5003 ts_tablet_manager.cc:1905] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:13:11 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:11.182173 5004 ts_tablet_manager.cc:1918] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:11.182372 5004 log.cc:1196] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/7d749d134b724a929514363513ff854b
I20260927 05:13:11.182492 5003 ts_tablet_manager.cc:1918] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:11.182672 5003 log.cc:1196] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/7d749d134b724a929514363513ff854b
I20260927 05:13:11.182950 5004 ts_tablet_manager.cc:1939] T 7d749d134b724a929514363513ff854b P dffcaabdd22e438289cdc45c1cf0ffdb: Deleting consensus metadata
I20260927 05:13:11.183235 5003 ts_tablet_manager.cc:1939] T 7d749d134b724a929514363513ff854b P d0e2019932d14b42b8299fed0b505f31: Deleting consensus metadata
I20260927 05:13:11.184697 4493 catalog_manager.cc:4841] TS dffcaabdd22e438289cdc45c1cf0ffdb (127.0.97.1:37533): tablet 7d749d134b724a929514363513ff854b (table test-table [id=215d9148177645a6b1b65c1b5d3d6827]) successfully deleted
I20260927 05:13:11.184831 4492 catalog_manager.cc:4841] TS d0e2019932d14b42b8299fed0b505f31 (127.0.97.3:34329): tablet 7d749d134b724a929514363513ff854b (table test-table [id=215d9148177645a6b1b65c1b5d3d6827]) successfully deleted
I20260927 05:13:11.188781 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.160693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.97.1:37533)
0927 05:13:11.160858 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.160863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.171469 (+ 10606us) server_negotiation.cc:407] Connection header received
0927 05:13:11.171656 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.171663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.171778 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.171891 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:11.173671 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.174775 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.176575 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.177204 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.177332 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.182202 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:11.182217 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:11.182237 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:11.182286 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:11.186044 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.186671 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.186680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.186683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.186878 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.187332 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.187337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.187340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.187671 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:11.187992 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.188204 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.188404 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:13:11.189384 4689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:11.312134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4567
I20260927 05:13:11.329085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4700
Sep 27 05:13:11 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:13:11.345322 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.329858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.0.97.2:38999)
0927 05:13:11.330039 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.330046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.330066 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:13:11.330228 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.330234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.330341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.330459 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:11.331649 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.332767 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.333999 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.334650 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.334771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.339672 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:11.339685 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:11.339700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:11.339736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:11.342770 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.343516 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.343523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.343525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.343666 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.344014 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.344017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.344019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.344312 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:11.344542 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.344856 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.344997 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:13:11.346014 4822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:11.454017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4700
I20260927 05:13:11.472020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4832
Sep 27 05:13:11 dist-test-slave-1t6p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:13:11.486173 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:11.472764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.97.3:34329)
0927 05:13:11.472963 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:11.472971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:11.472993 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:13:11.473199 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:11.473203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:11.473265 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:11.473367 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:11.474549 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.475319 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.476521 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:11.476919 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:11.477051 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:11.481291 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:11.481310 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:11.481324 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:11.481356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:11.483916 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.484468 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.484474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.484476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.484601 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:11.484900 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:11.484903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:11.484905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:11.485240 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:11.485441 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:11.485662 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 05:13:11.485821 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:11.486773 4954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:11.572815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4832
I20260927 05:13:11.589131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 4475
I20260927 05:13:11.590174 4536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:11.697121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 4475
2026-09-27T05:13:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4274 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T05:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:11Z Disabled control of system clock
I20260927 05:13:11.760217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37139
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 05:13:12.066546 5013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:12.066793 5013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:12.066872 5013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:12.073619 5013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:12.073705 5013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:12.073741 5013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:12.073796 5013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:12.073836 5013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:12.082430 5013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:12.083619 5013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:12.085273 5013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:12.096951 5018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:12.096971 5019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:12.097470 5013 server_base.cc:1034] running on GCE node
W20260927 05:13:12.097678 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:12.098454 5013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:12.099817 5013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:12.101069 5013 hybrid_clock.cc:648] HybridClock initialized: now 1790485992101038 us; error 42 us; skew 500 ppm
I20260927 05:13:12.101430 5013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:12.103642 5013 webserver.cc:460] Webserver started at http://127.0.97.62:44273/ using document root <none> and password file <none>
I20260927 05:13:12.104300 5013 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:12.104430 5013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:12.104744 5013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:12.107546 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "6e7a80237990494c974fc5680a2ce30c"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "eec0b041cbb1901783083fcb39f6025d"
server_key_iv: "b909354d4f84dbee4187ca7240e9c7c3"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.108309 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e7a80237990494c974fc5680a2ce30c"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "eec0b041cbb1901783083fcb39f6025d"
server_key_iv: "b909354d4f84dbee4187ca7240e9c7c3"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.113058 5013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:12.116250 5027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:12.117483 5013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:12.117651 5013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "6e7a80237990494c974fc5680a2ce30c"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "eec0b041cbb1901783083fcb39f6025d"
server_key_iv: "b909354d4f84dbee4187ca7240e9c7c3"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.117859 5013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:12.123212 5013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:12.123893 5013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:12.124187 5013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:12.141209 5013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37139
I20260927 05:13:12.141247 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37139 every 8 connection(s)
I20260927 05:13:12.142411 5013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:12.144608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5013
I20260927 05:13:12.144815 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:12.145609 388 external_mini_cluster.cc:1428] Setting key c4ea9a6be19bba3da92215e113dc2877
I20260927 05:13:12.146453 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:12.154078 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:12.147593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.0.97.62:37139)
0927 05:13:12.148544 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:12.148554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:12.148591 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:13:12.150746 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:12.150769 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:12.151234 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:12.151543 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:12.152590 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:12.152599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:12.152611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:12.152624 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:12.152681 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:12.152698 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:12.152862 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:12.152971 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 05:13:12.153240 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":388,"threads_started":1}
I20260927 05:13:12.158494 5079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: Bootstrap starting.
I20260927 05:13:12.161213 5079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:12.162159 5079 log.cc:824] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:12.164649 5079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: No bootstrap required, opened a new log
I20260927 05:13:12.171340 5079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } }
I20260927 05:13:12.171689 5079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:12.171789 5079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7a80237990494c974fc5680a2ce30c, State: Initialized, Role: FOLLOWER
I20260927 05:13:12.172369 5079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } }
I20260927 05:13:12.172679 5079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:12.172823 5079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:12.172992 5079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:12.174933 5079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } }
I20260927 05:13:12.175446 5079 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e7a80237990494c974fc5680a2ce30c; no voters:
I20260927 05:13:12.176010 5079 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:12.176124 5084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:12.176864 5084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [term 1 LEADER]: Becoming Leader. State: Replica: 6e7a80237990494c974fc5680a2ce30c, State: Running, Role: LEADER
I20260927 05:13:12.177430 5084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } }
I20260927 05:13:12.178047 5079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:12.179896 5086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7a80237990494c974fc5680a2ce30c. Latest consensus state: current_term: 1 leader_uuid: "6e7a80237990494c974fc5680a2ce30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } } }
I20260927 05:13:12.180234 5086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:12.181211 5090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:12.181327 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e7a80237990494c974fc5680a2ce30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7a80237990494c974fc5680a2ce30c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37139 } } }
I20260927 05:13:12.181805 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:12.187400 5090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:12.195968 5090 catalog_manager.cc:1347] Generated new cluster ID: e842f0c431524b0a96d4c4cad691bfc6
I20260927 05:13:12.196069 5090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:12.205109 5090 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:12.205500 5090 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:12.227217 5090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: Generated new TSK 0
I20260927 05:13:12.228009 5090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:13:12.237303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--builtin_ntp_servers=127.0.97.20:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 05:13:12.545056 5103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:12.545305 5103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:12.545389 5103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:12.552443 5103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:12.552536 5103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:12.552647 5103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:12.561085 5103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:12.562119 5103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:12.563767 5103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:12.575956 5108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:12.576061 5111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:12.576104 5109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:12.576891 5103 server_base.cc:1034] running on GCE node
I20260927 05:13:12.577432 5103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:12.578420 5103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:12.579630 5103 hybrid_clock.cc:648] HybridClock initialized: now 1790485992579586 us; error 60 us; skew 500 ppm
I20260927 05:13:12.579924 5103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:12.581938 5103 webserver.cc:460] Webserver started at http://127.0.97.1:41773/ using document root <none> and password file <none>
I20260927 05:13:12.582597 5103 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:12.582721 5103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:12.583076 5103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:12.585778 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "422f70c55a7845829c216b71a84c2124"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "802f97f88b7acb91a505a28ccda6cb1c"
server_key_iv: "b38c469bd355e0188d2379e8e000ea95"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.586563 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "422f70c55a7845829c216b71a84c2124"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "802f97f88b7acb91a505a28ccda6cb1c"
server_key_iv: "b38c469bd355e0188d2379e8e000ea95"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.591310 5103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 05:13:12.594475 5117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:12.595786 5103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:12.595954 5103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "422f70c55a7845829c216b71a84c2124"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "802f97f88b7acb91a505a28ccda6cb1c"
server_key_iv: "b38c469bd355e0188d2379e8e000ea95"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.596179 5103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:12.601387 5103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:12.602111 5103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:12.602363 5103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:12.603341 5103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:12.604643 5103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:12.604763 5103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:12.604887 5103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:12.604971 5103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:12.636345 5103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38311
I20260927 05:13:12.636426 5229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38311 every 8 connection(s)
I20260927 05:13:12.637672 5103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:12.643723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5103
I20260927 05:13:12.643939 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:12.644531 388 external_mini_cluster.cc:1428] Setting key aa05bdd2a150e1bb8f2f88a6e78ce136
I20260927 05:13:12.645259 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:12.640121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52419 (local address 127.0.97.62:37139)
0927 05:13:12.640280 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:12.640284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:12.641495 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 05:13:12.643111 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:12.643116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:12.643182 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:12.643287 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:12.644547 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:12.644552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:12.644557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:12.644575 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:12.644612 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:12.644643 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:12.644749 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:12.644943 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 05:13:12.645007 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:13:12.646101 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:12.640519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37139 (local address 127.0.97.1:52419)
0927 05:13:12.641356 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:12.641389 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:12.642829 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:12.643512 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:12.643519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:12.643600 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:12.644272 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:12.644274 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:12.644309 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:12.644382 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:12.644722 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:12.644731 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:12.644738 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:12.645012 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 05:13:12.645232 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260927 05:13:12.647347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--builtin_ntp_servers=127.0.97.20:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 05:13:12.647609 5230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37139
I20260927 05:13:12.647863 5230 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:12.648363 5230 heartbeater.cc:511] Master 127.0.97.62:37139 requested a full tablet report, sending...
I20260927 05:13:12.650580 5044 ts_manager.cc:194] Registered new tserver with Master: 422f70c55a7845829c216b71a84c2124 (127.0.97.1:38311)
W20260927 05:13:12.955232 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:12.955485 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:12.955646 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:12.962518 5233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:12.962608 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:12.962812 5233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:12.971477 5233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:12.972576 5233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:12.974331 5233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:12.987035 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:12.987030 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:12.987118 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:12.988044 5233 server_base.cc:1034] running on GCE node
I20260927 05:13:12.988629 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:12.989579 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:12.990800 5233 hybrid_clock.cc:648] HybridClock initialized: now 1790485992990761 us; error 61 us; skew 500 ppm
I20260927 05:13:12.991173 5233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:12.993352 5233 webserver.cc:460] Webserver started at http://127.0.97.2:35929/ using document root <none> and password file <none>
I20260927 05:13:12.994201 5233 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:12.994349 5233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:12.994694 5233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:12.998072 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "52bd8f7d870a415bbdcd28020b2adcb9"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "95b521ee298bab789db8e5ddd8ad6a69"
server_key_iv: "5e6e10542c793bd3b36981c71efff5a0"
server_key_version: "encryptionkey@0"
I20260927 05:13:12.998826 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "52bd8f7d870a415bbdcd28020b2adcb9"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "95b521ee298bab789db8e5ddd8ad6a69"
server_key_iv: "5e6e10542c793bd3b36981c71efff5a0"
server_key_version: "encryptionkey@0"
I20260927 05:13:13.003577 5233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:13:13.006649 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.007979 5233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:13.008142 5233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "52bd8f7d870a415bbdcd28020b2adcb9"
format_stamp: "Formatted at 2026-09-27 05:13:12 on dist-test-slave-1t6p"
server_key: "95b521ee298bab789db8e5ddd8ad6a69"
server_key_iv: "5e6e10542c793bd3b36981c71efff5a0"
server_key_version: "encryptionkey@0"
I20260927 05:13:13.008347 5233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:13.013548 5233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:13.014249 5233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:13.014509 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:13.015419 5233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:13.016664 5233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:13.016770 5233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.016896 5233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:13.016991 5233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.047828 5233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42665
I20260927 05:13:13.047919 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42665 every 8 connection(s)
I20260927 05:13:13.049116 5233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:13.055341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5233
I20260927 05:13:13.055555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:13.055946 388 external_mini_cluster.cc:1428] Setting key bf9f0bc403a18152b792cff7f2874043
I20260927 05:13:13.057534 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.051707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58119 (local address 127.0.97.62:37139)
0927 05:13:13.051869 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.051874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.053233 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 05:13:13.054929 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.054933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.055054 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.055176 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.056547 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.056553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.056557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.056576 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.056615 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.056631 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.056749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.057227 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.057279 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:13:13.058605 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.052209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37139 (local address 127.0.97.2:58119)
0927 05:13:13.053089 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.053130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.054707 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.055345 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.055352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.055434 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.056190 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.056193 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.056242 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.056345 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.056953 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.056963 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.056970 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.057270 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.057505 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":265,"threads_started":1}
I20260927 05:13:13.059111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--builtin_ntp_servers=127.0.97.20:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 05:13:13.059867 5360 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37139
I20260927 05:13:13.060182 5360 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:13.060808 5360 heartbeater.cc:511] Master 127.0.97.62:37139 requested a full tablet report, sending...
I20260927 05:13:13.062757 5044 ts_manager.cc:194] Registered new tserver with Master: 52bd8f7d870a415bbdcd28020b2adcb9 (127.0.97.2:42665)
W20260927 05:13:13.371091 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:13.371366 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:13.371531 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:13.378408 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:13.378497 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:13.378597 5363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:13.387112 5363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:13.388288 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:13.389847 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:13.402436 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:13.402390 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:13.402482 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:13.404281 5363 server_base.cc:1034] running on GCE node
I20260927 05:13:13.404810 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:13.405632 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:13.406848 5363 hybrid_clock.cc:648] HybridClock initialized: now 1790485993406820 us; error 45 us; skew 500 ppm
I20260927 05:13:13.407207 5363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:13.409274 5363 webserver.cc:460] Webserver started at http://127.0.97.3:36207/ using document root <none> and password file <none>
I20260927 05:13:13.409884 5363 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:13.410014 5363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:13.410306 5363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:13.413024 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "4f5ece859e8b495d988034c47d2388f2"
format_stamp: "Formatted at 2026-09-27 05:13:13 on dist-test-slave-1t6p"
server_key: "c9235a7b169a2779664cde744bba2927"
server_key_iv: "44a958609a99ac4b616d73d383efb85b"
server_key_version: "encryptionkey@0"
I20260927 05:13:13.413779 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4f5ece859e8b495d988034c47d2388f2"
format_stamp: "Formatted at 2026-09-27 05:13:13 on dist-test-slave-1t6p"
server_key: "c9235a7b169a2779664cde744bba2927"
server_key_iv: "44a958609a99ac4b616d73d383efb85b"
server_key_version: "encryptionkey@0"
I20260927 05:13:13.418509 5363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:13:13.421684 5377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.422855 5363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:13:13.423048 5363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "4f5ece859e8b495d988034c47d2388f2"
format_stamp: "Formatted at 2026-09-27 05:13:13 on dist-test-slave-1t6p"
server_key: "c9235a7b169a2779664cde744bba2927"
server_key_iv: "44a958609a99ac4b616d73d383efb85b"
server_key_version: "encryptionkey@0"
I20260927 05:13:13.423271 5363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:13.428388 5363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:13.429085 5363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:13.429349 5363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:13.430161 5363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:13.431458 5363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:13.431577 5363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.431708 5363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:13.431797 5363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:13.463194 5363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40257
I20260927 05:13:13.463236 5489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40257 every 8 connection(s)
I20260927 05:13:13.464407 5363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:13.468987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5363
I20260927 05:13:13.469154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:13.469563 388 external_mini_cluster.cc:1428] Setting key e30970513cb00d534c66f45e6190030d
I20260927 05:13:13.472352 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.466828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43883 (local address 127.0.97.62:37139)
0927 05:13:13.467062 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.467068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.468344 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 05:13:13.470102 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.470108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.470175 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.470294 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.471648 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.471654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.471660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.471682 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.471710 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.471722 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.471842 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.472038 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.472101 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:13:13.473187 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.467352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37139 (local address 127.0.97.3:43883)
0927 05:13:13.468195 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.468227 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.469841 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.470507 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.470514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.470596 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.471320 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.471323 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.471377 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.471454 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.471825 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.471835 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.471842 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.472117 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.472343 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
I20260927 05:13:13.474270 5490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37139
I20260927 05:13:13.474542 5490 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:13.475132 5490 heartbeater.cc:511] Master 127.0.97.62:37139 requested a full tablet report, sending...
I20260927 05:13:13.476315 5044 ts_manager.cc:194] Registered new tserver with Master: 4f5ece859e8b495d988034c47d2388f2 (127.0.97.3:40257)
I20260927 05:13:13.484440 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:13:13.490967 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.488751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.0.97.62:37139)
0927 05:13:13.488894 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.488898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.489439 (+ 541us) server_negotiation.cc:407] Connection header received
0927 05:13:13.489596 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.489600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.489648 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.489733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.490516 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.490520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.490523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.490537 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.490564 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.490575 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.490667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.490733 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.490772 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:13:13.494426 5044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:13.501132 5044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:13.525830 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.512049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42665 (local address 127.0.0.1:41392)
0927 05:13:13.522703 (+ 10654us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.522720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.522837 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.523457 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.523465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.523570 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.524553 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.524555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.524578 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.524632 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.525016 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.525026 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.525033 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.525188 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.525267 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10544,"thread_start_us":11090,"threads_started":1}
I20260927 05:13:13.526307 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.511943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.97.1:38311)
0927 05:13:13.523077 (+ 11134us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.523087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.523122 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:13:13.523196 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.523206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.523437 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.523579 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.525185 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.525193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.525205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.525222 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.525277 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.525296 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.525426 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.525934 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.526037 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10963,"thread_start_us":10861,"threads_started":1}
I20260927 05:13:13.526665 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.512673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.0.97.2:42665)
0927 05:13:13.513167 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.513178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.522820 (+ 9642us) server_negotiation.cc:407] Connection header received
0927 05:13:13.522977 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.522990 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.523279 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.523452 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.524785 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.524793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.524807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.524824 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.524884 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.524907 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.526127 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.526206 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.526299 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":245,"threads_started":1}
I20260927 05:13:13.526453 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.511801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38311 (local address 127.0.0.1:49944)
0927 05:13:13.516332 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.516386 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.516549 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.523843 (+ 7294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.523847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.523869 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.524936 (+ 1067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.524938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.524953 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.525016 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.525474 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.525485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.525492 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.525625 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.525682 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4227,"thread_start_us":4039,"threads_started":1}
I20260927 05:13:13.528259 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.512402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40257 (local address 127.0.0.1:35064)
0927 05:13:13.523364 (+ 10962us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.523385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.523479 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.524806 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.524810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.524830 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.525537 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.525541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.525555 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.525592 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.526151 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.526156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.526161 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.527756 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.527810 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10863,"thread_start_us":10092,"threads_started":1}
I20260927 05:13:13.528463 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.512688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.97.3:40257)
0927 05:13:13.515671 (+ 2983us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.515682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.523596 (+ 7914us) server_negotiation.cc:407] Connection header received
0927 05:13:13.523668 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.523675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.523862 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.524002 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.525769 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.525778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.525791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.525831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.525904 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.525929 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.526095 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.526391 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.526468 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2773,"thread_start_us":2572,"threads_started":1}
I20260927 05:13:13.530082 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 (DEFAULT_TABLE table=test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:13.530092 5165 tablet_service.cc:1467] Processing CreateTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 (DEFAULT_TABLE table=test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:13.531387 5165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41880b9c52f44e16bfbdb69aaf457b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:13.531430 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41880b9c52f44e16bfbdb69aaf457b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:13.533380 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 (DEFAULT_TABLE table=test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:13.534610 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41880b9c52f44e16bfbdb69aaf457b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:13.541741 5508 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Bootstrap starting.
I20260927 05:13:13.545051 5509 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Bootstrap starting.
I20260927 05:13:13.545681 5508 tablet_bootstrap.cc:654] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:13.546947 5508 log.cc:824] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:13.548308 5510 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Bootstrap starting.
I20260927 05:13:13.548576 5509 tablet_bootstrap.cc:654] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:13.549717 5509 log.cc:824] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:13.549780 5508 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: No bootstrap required, opened a new log
I20260927 05:13:13.550029 5508 ts_tablet_manager.cc:1397] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 05:13:13.551640 5510 tablet_bootstrap.cc:654] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:13.552564 5509 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: No bootstrap required, opened a new log
I20260927 05:13:13.552796 5509 ts_tablet_manager.cc:1397] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:13:13.553164 5510 log.cc:824] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:13.555903 5510 tablet_bootstrap.cc:492] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: No bootstrap required, opened a new log
I20260927 05:13:13.556129 5510 ts_tablet_manager.cc:1397] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:13:13.558025 5508 raft_consensus.cc:348] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.558360 5508 raft_consensus.cc:374] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:13.558480 5508 raft_consensus.cc:729] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422f70c55a7845829c216b71a84c2124, State: Initialized, Role: FOLLOWER
I20260927 05:13:13.559128 5508 consensus_queue.cc:260] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.562726 5509 raft_consensus.cc:348] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.563156 5509 raft_consensus.cc:374] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:13.563241 5509 raft_consensus.cc:729] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd8f7d870a415bbdcd28020b2adcb9, State: Initialized, Role: FOLLOWER
I20260927 05:13:13.563858 5509 consensus_queue.cc:260] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.564014 5230 heartbeater.cc:503] Master 127.0.97.62:37139 was elected leader, sending a full tablet report...
I20260927 05:13:13.564692 5508 ts_tablet_manager.cc:1428] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:13:13.564787 5360 heartbeater.cc:503] Master 127.0.97.62:37139 was elected leader, sending a full tablet report...
I20260927 05:13:13.566413 5509 ts_tablet_manager.cc:1428] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 05:13:13.567369 5510 raft_consensus.cc:348] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.567679 5510 raft_consensus.cc:374] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:13.567792 5510 raft_consensus.cc:729] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5ece859e8b495d988034c47d2388f2, State: Initialized, Role: FOLLOWER
I20260927 05:13:13.568337 5510 consensus_queue.cc:260] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.575250 5490 heartbeater.cc:503] Master 127.0.97.62:37139 was elected leader, sending a full tablet report...
I20260927 05:13:13.576061 5510 ts_tablet_manager.cc:1428] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
W20260927 05:13:13.639811 5231 tablet.cc:2369] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:13.716156 5491 tablet.cc:2369] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:13.801106 5361 tablet.cc:2369] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:13.852702 5515 raft_consensus.cc:482] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:13.852950 5515 raft_consensus.cc:504] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.854463 5515 leader_election.cc:288] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422f70c55a7845829c216b71a84c2124 (127.0.97.1:38311), 4f5ece859e8b495d988034c47d2388f2 (127.0.97.3:40257)
I20260927 05:13:13.858275 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.854979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60187 (local address 127.0.97.3:40257)
0927 05:13:13.855229 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.855237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.856357 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 05:13:13.856542 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.856552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.856600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.856700 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.857681 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.857687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.857692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.857710 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.857748 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.857759 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.857881 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.857968 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.858017 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 05:13:13.858444 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.854710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40257 (local address 127.0.97.2:60187)
0927 05:13:13.856246 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.856261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.856409 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.856720 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.856725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.856748 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.857456 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.857462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.857481 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.857529 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.857848 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.857853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.857859 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.857958 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.858008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"thread_start_us":1184,"threads_started":1}
I20260927 05:13:13.858484 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.854868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41393 (local address 127.0.97.1:38311)
0927 05:13:13.855130 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.855140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.856427 (+ 1287us) server_negotiation.cc:407] Connection header received
0927 05:13:13.856591 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.856596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.856652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.856754 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.857833 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.857839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.857843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.857860 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.857904 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.857914 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.858033 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.858155 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.858214 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:13:13.858650 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.854719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38311 (local address 127.0.97.2:41393)
0927 05:13:13.856340 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:13.856357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:13.856447 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:13.856771 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:13.856775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:13.856797 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:13.857647 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:13:13.857650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:13.857661 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:13:13.857692 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:13.858013 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:13.858018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:13.858023 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:13:13.858178 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:13:13.858221 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":1180,"threads_started":1}
I20260927 05:13:13.859683 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41880b9c52f44e16bfbdb69aaf457b69" candidate_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5ece859e8b495d988034c47d2388f2" is_pre_election: true
I20260927 05:13:13.859683 5185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41880b9c52f44e16bfbdb69aaf457b69" candidate_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422f70c55a7845829c216b71a84c2124" is_pre_election: true
I20260927 05:13:13.860231 5445 raft_consensus.cc:2454] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52bd8f7d870a415bbdcd28020b2adcb9 in term 0.
I20260927 05:13:13.860230 5185 raft_consensus.cc:2454] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52bd8f7d870a415bbdcd28020b2adcb9 in term 0.
I20260927 05:13:13.860868 5249 leader_election.cc:302] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422f70c55a7845829c216b71a84c2124, 52bd8f7d870a415bbdcd28020b2adcb9; no voters:
I20260927 05:13:13.861228 5515 raft_consensus.cc:2781] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:13.861375 5515 raft_consensus.cc:482] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:13.861496 5515 raft_consensus.cc:3037] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:13.863628 5515 raft_consensus.cc:504] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.864257 5515 leader_election.cc:288] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [CANDIDATE]: Term 1 election: Requested vote from peers 422f70c55a7845829c216b71a84c2124 (127.0.97.1:38311), 4f5ece859e8b495d988034c47d2388f2 (127.0.97.3:40257)
I20260927 05:13:13.864593 5185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41880b9c52f44e16bfbdb69aaf457b69" candidate_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422f70c55a7845829c216b71a84c2124"
I20260927 05:13:13.864702 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41880b9c52f44e16bfbdb69aaf457b69" candidate_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5ece859e8b495d988034c47d2388f2"
I20260927 05:13:13.864785 5185 raft_consensus.cc:3037] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:13.864876 5445 raft_consensus.cc:3037] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:13.867959 5445 raft_consensus.cc:2454] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bd8f7d870a415bbdcd28020b2adcb9 in term 1.
I20260927 05:13:13.867975 5185 raft_consensus.cc:2454] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bd8f7d870a415bbdcd28020b2adcb9 in term 1.
I20260927 05:13:13.868335 5250 leader_election.cc:302] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5ece859e8b495d988034c47d2388f2, 52bd8f7d870a415bbdcd28020b2adcb9; no voters:
I20260927 05:13:13.868615 5515 raft_consensus.cc:2781] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:13.869117 5515 raft_consensus.cc:686] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 1 LEADER]: Becoming Leader. State: Replica: 52bd8f7d870a415bbdcd28020b2adcb9, State: Running, Role: LEADER
I20260927 05:13:13.869561 5515 consensus_queue.cc:237] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } }
I20260927 05:13:13.874027 5043 catalog_manager.cc:5495] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52bd8f7d870a415bbdcd28020b2adcb9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52bd8f7d870a415bbdcd28020b2adcb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:13.971911 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:13.969750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.97.2:42665)
0927 05:13:13.969993 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:13.969999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:13.970018 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:13:13.970227 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:13.970231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:13.970305 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:13.970422 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:13.971328 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:13.971334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:13.971338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:13.971356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:13.971396 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:13.971407 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:13.971530 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:13.971609 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 05:13:13.971665 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 05:13:13.979722 5445 raft_consensus.cc:1261] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 1 FOLLOWER]: Refusing update from remote peer 52bd8f7d870a415bbdcd28020b2adcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:13.979724 5185 raft_consensus.cc:1261] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 1 FOLLOWER]: Refusing update from remote peer 52bd8f7d870a415bbdcd28020b2adcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:13.980592 5519 consensus_queue.cc:1035] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5ece859e8b495d988034c47d2388f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:13.980901 5515 consensus_queue.cc:1035] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422f70c55a7845829c216b71a84c2124" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:13.991904 5522 mvcc.cc:205] Tried to move back new op lower bound from 7333830631331131392 to 7333830630893019136. Current Snapshot: MvccSnapshot[applied={T|T < 7333830631331131392}]
I20260927 05:13:13.996065 5524 mvcc.cc:205] Tried to move back new op lower bound from 7333830631331131392 to 7333830630893019136. Current Snapshot: MvccSnapshot[applied={T|T < 7333830631331131392}]
I20260927 05:13:14.008893 5044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:14.009114 5044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:14.012377 5044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e7a80237990494c974fc5680a2ce30c: Sending DeleteTablet for 3 replicas of tablet 41880b9c52f44e16bfbdb69aaf457b69
I20260927 05:13:14.013463 5165 tablet_service.cc:1514] Processing DeleteTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:14 UTC) from {username='slave'} at 127.0.0.1:49944
I20260927 05:13:14.013543 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:14 UTC) from {username='slave'} at 127.0.0.1:35064
I20260927 05:13:14.013578 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet 41880b9c52f44e16bfbdb69aaf457b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:14 UTC) from {username='slave'} at 127.0.0.1:41392
I20260927 05:13:14.015190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5103
I20260927 05:13:14.026649 5532 tablet_replica.cc:330] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: stopping tablet replica
I20260927 05:13:14.027258 5532 raft_consensus.cc:2229] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:14.027944 5532 raft_consensus.cc:2258] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:14.030081 5532 ts_tablet_manager.cc:1905] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:14.038461 5532 ts_tablet_manager.cc:1918] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:14.038698 5532 log.cc:1196] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/41880b9c52f44e16bfbdb69aaf457b69
I20260927 05:13:14.039348 5532 ts_tablet_manager.cc:1939] T 41880b9c52f44e16bfbdb69aaf457b69 P 422f70c55a7845829c216b71a84c2124: Deleting consensus metadata
I20260927 05:13:14.041297 5029 catalog_manager.cc:4841] TS 422f70c55a7845829c216b71a84c2124 (127.0.97.1:38311): tablet 41880b9c52f44e16bfbdb69aaf457b69 (table test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]) successfully deleted
I20260927 05:13:14.043543 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:14.018271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.0.97.1:38311)
0927 05:13:14.018459 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:14.018468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:14.040867 (+ 22399us) server_negotiation.cc:407] Connection header received
0927 05:13:14.041337 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:14.041344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:14.041454 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:14.041528 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:14.042646 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:14.042655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:14.042663 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:14.042688 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:14.042734 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:14.042754 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:14.042893 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:14.043170 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 05:13:14.043238 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:13:14.043583 5534 tablet_replica.cc:330] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: stopping tablet replica
I20260927 05:13:14.044425 5225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:14.044143 5534 raft_consensus.cc:2229] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:14.050855 5534 raft_consensus.cc:2258] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:14.051949 5533 tablet_replica.cc:330] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: stopping tablet replica
I20260927 05:13:14.052589 5533 raft_consensus.cc:2229] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:14.053501 5533 raft_consensus.cc:2258] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:14.054320 5534 ts_tablet_manager.cc:1905] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:14.054905 5533 ts_tablet_manager.cc:1905] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:14.063575 5533 ts_tablet_manager.cc:1918] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:14.063778 5533 log.cc:1196] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/41880b9c52f44e16bfbdb69aaf457b69
I20260927 05:13:14.064332 5533 ts_tablet_manager.cc:1939] T 41880b9c52f44e16bfbdb69aaf457b69 P 52bd8f7d870a415bbdcd28020b2adcb9: Deleting consensus metadata
I20260927 05:13:14.066030 5031 catalog_manager.cc:4841] TS 52bd8f7d870a415bbdcd28020b2adcb9 (127.0.97.2:42665): tablet 41880b9c52f44e16bfbdb69aaf457b69 (table test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]) successfully deleted
I20260927 05:13:14.073278 5534 ts_tablet_manager.cc:1918] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:14.073536 5534 log.cc:1196] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/41880b9c52f44e16bfbdb69aaf457b69
I20260927 05:13:14.074149 5534 ts_tablet_manager.cc:1939] T 41880b9c52f44e16bfbdb69aaf457b69 P 4f5ece859e8b495d988034c47d2388f2: Deleting consensus metadata
I20260927 05:13:14.076002 5030 catalog_manager.cc:4841] TS 4f5ece859e8b495d988034c47d2388f2 (127.0.97.3:40257): tablet 41880b9c52f44e16bfbdb69aaf457b69 (table test-table [id=99a76ba99ffb4cdf9ac51a66795bcaef]) successfully deleted
I20260927 05:13:14.156334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5103
I20260927 05:13:14.179387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5233
I20260927 05:13:14.182330 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:14.180291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.0.97.2:42665)
0927 05:13:14.180485 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:14.180492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:14.180514 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:13:14.180600 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:14.180604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:14.180651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:14.180755 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:14.181683 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:14.181688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:14.181693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:14.181713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:14.181748 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:14.181759 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:14.181889 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:14.182016 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 05:13:14.182068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:13:14.183221 5355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:14.289872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5233
I20260927 05:13:14.308008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5363
I20260927 05:13:14.310480 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:14.308707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.97.3:40257)
0927 05:13:14.308867 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:14.308871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:14.308936 (+ 65us) server_negotiation.cc:407] Connection header received
0927 05:13:14.309087 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:14.309090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:14.309171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:14.309264 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:14.310002 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:14.310006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:14.310009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:14.310023 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:14.310051 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:14.310061 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:14.310159 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:14.310220 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 05:13:14.310262 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:13:14.311084 5485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:14.395358 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5363
I20260927 05:13:14.411051 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5013
I20260927 05:13:14.412161 5074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:14.514474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5013
2026-09-27T05:13:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2816 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 05:13:14.554566 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:13:14.557176 388 webserver.cc:460] Webserver started at https://127.0.0.1:45911/ using document root <none> and password file <none>
I20260927 05:13:14.557456 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:13:14.558076 388 webserver.cc:460] Webserver started at http://127.0.0.1:33409/ using document root <none> and password file <none>
2026-09-27T05:13:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:14Z Disabled control of system clock
I20260927 05:13:14.581876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37703
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37703
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45911/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/testchainca.cert with env {}
W20260927 05:13:14.887359 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:14.887683 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:14.887838 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:14.895521 5545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:14.895649 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:14.895712 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:14.895792 5545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:14.895889 5545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:13:14.895972 5545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:13:14.896047 5545 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45911/jwks.json
I20260927 05:13:14.905517 5545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45911/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:14.906771 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:14.908077 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:14.919229 5550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:14.919193 5551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:14.919262 5553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:14.920390 5545 server_base.cc:1034] running on GCE node
I20260927 05:13:14.921131 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:14.922345 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:14.923566 5545 hybrid_clock.cc:648] HybridClock initialized: now 1790485994923537 us; error 44 us; skew 500 ppm
I20260927 05:13:14.923893 5545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:14.929244 5545 webserver.cc:460] Webserver started at http://127.0.97.62:38301/ using document root <none> and password file <none>
I20260927 05:13:14.929900 5545 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:14.930032 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:14.930335 5545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:14.933100 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "aafbb910165e4ce9a1474455c8591e14"
format_stamp: "Formatted at 2026-09-27 05:13:14 on dist-test-slave-1t6p"
server_key: "c195d2ddd2f8a2626bd60d011aa920af"
server_key_iv: "52da5c81cb599efceaa1a2175be4cca3"
server_key_version: "encryptionkey@0"
I20260927 05:13:14.933876 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "aafbb910165e4ce9a1474455c8591e14"
format_stamp: "Formatted at 2026-09-27 05:13:14 on dist-test-slave-1t6p"
server_key: "c195d2ddd2f8a2626bd60d011aa920af"
server_key_iv: "52da5c81cb599efceaa1a2175be4cca3"
server_key_version: "encryptionkey@0"
I20260927 05:13:14.938530 5545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 05:13:14.941715 5559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:14.942886 5545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:14.943085 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "aafbb910165e4ce9a1474455c8591e14"
format_stamp: "Formatted at 2026-09-27 05:13:14 on dist-test-slave-1t6p"
server_key: "c195d2ddd2f8a2626bd60d011aa920af"
server_key_iv: "52da5c81cb599efceaa1a2175be4cca3"
server_key_version: "encryptionkey@0"
I20260927 05:13:14.943297 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:14.957273 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:14.958050 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:14.958324 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:14.976094 5545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37703
I20260927 05:13:14.976114 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37703 every 8 connection(s)
I20260927 05:13:14.977391 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:14.980962 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:14.986557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5545
I20260927 05:13:14.986761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:14.987452 388 external_mini_cluster.cc:1428] Setting key ebbff8f7f8d2884841fc272b30830a85
I20260927 05:13:14.991269 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14: Bootstrap starting.
I20260927 05:13:14.994433 5611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:14.995639 5611 log.cc:824] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:14.998061 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14: No bootstrap required, opened a new log
I20260927 05:13:15.000735 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:14.989341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.97.62:37703)
0927 05:13:14.990309 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:14.990319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:14.990359 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:13:14.992098 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:14.992120 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:14.992556 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:14.992863 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:14.993505 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:14.994790 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:14.996571 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:14.997191 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:14.997381 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:14.998558 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:14.998565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:14.998578 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:14.999136 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:14.999208 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:14.999228 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:14.999353 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:14.999588 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 05:13:14.999854 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":366,"threads_started":1}
I20260927 05:13:15.005206 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:13:15.007668 5611 raft_consensus.cc:348] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } }
I20260927 05:13:15.008085 5611 raft_consensus.cc:374] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:15.008194 5611 raft_consensus.cc:729] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aafbb910165e4ce9a1474455c8591e14, State: Initialized, Role: FOLLOWER
I20260927 05:13:15.008823 5611 consensus_queue.cc:260] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } }
I20260927 05:13:15.009088 5611 raft_consensus.cc:388] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:15.009222 5611 raft_consensus.cc:482] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:15.009372 5611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:15.011461 5611 raft_consensus.cc:504] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } }
I20260927 05:13:15.011930 5611 leader_election.cc:302] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aafbb910165e4ce9a1474455c8591e14; no voters:
I20260927 05:13:15.012548 5611 leader_election.cc:288] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:15.012681 5616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:15.013314 5616 raft_consensus.cc:686] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [term 1 LEADER]: Becoming Leader. State: Replica: aafbb910165e4ce9a1474455c8591e14, State: Running, Role: LEADER
I20260927 05:13:15.013885 5616 consensus_queue.cc:237] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } }
I20260927 05:13:15.014631 5611 sys_catalog.cc:564] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:15.016832 5617 sys_catalog.cc:455] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aafbb910165e4ce9a1474455c8591e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } } }
I20260927 05:13:15.016808 5618 sys_catalog.cc:455] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aafbb910165e4ce9a1474455c8591e14. Latest consensus state: current_term: 1 leader_uuid: "aafbb910165e4ce9a1474455c8591e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafbb910165e4ce9a1474455c8591e14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37703 } } }
I20260927 05:13:15.017239 5618 sys_catalog.cc:458] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:15.021039 5617 sys_catalog.cc:458] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:15.022022 5622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:15.025874 5622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:15.033217 5622 catalog_manager.cc:1347] Generated new cluster ID: e02cd10a52664b15abd69f2b655858c6
I20260927 05:13:15.033334 5622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:15.044888 5622 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:15.046530 5622 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:15.057155 5622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aafbb910165e4ce9a1474455c8591e14: Generated new TSK 0
I20260927 05:13:15.057906 5622 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 05:13:15.095134 5641 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 05:13:15.095299 5641 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:13:15.096350 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:15.073867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.97.62:37703)
0927 05:13:15.074053 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:15.074059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:15.075620 (+ 1561us) server_negotiation.cc:407] Connection header received
0927 05:13:15.075867 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:15.075874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:15.075969 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:15.076124 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:13:15.076613 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.077719 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.079197 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.079871 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.080023 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:15.080464 (+ 441us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:13:15.095713 (+ 15249us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 05:13:15.096073 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 05:13:15.096467 5614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 05:13:15.097636 5640 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:15.073735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37703 (local address 127.0.0.1:40078)
0927 05:13:15.075516 (+ 1781us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:15.075536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:15.075664 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:15.076147 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:15.076150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:15.076159 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:13:15.076478 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:15.076486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.077848 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.077851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:15.079075 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:15.079085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.080010 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.080060 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:15.080176 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:15.080228 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:13:15.095974 (+ 15746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 05:13:15.096223 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":1601,"thread_start_us":1467,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:37703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 05:13:15.099383 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5545
I20260927 05:13:15.100504 5606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:15.207620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5545
2026-09-27T05:13:15Z chronyd exiting
I20260927 05:13:15.257117 388 test_util.cc:175] -----------------------------------------------
I20260927 05:13:15.257249 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485951045978-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 05:13:15.261032 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:13:15.262800 388 webserver.cc:460] Webserver started at https://127.0.0.1:45063/ using document root <none> and password file <none>
I20260927 05:13:15.263083 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:13:15.263689 388 webserver.cc:460] Webserver started at http://127.0.0.1:37953/ using document root <none> and password file <none>
2026-09-27T05:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:15Z Disabled control of system clock
I20260927 05:13:15.288163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45975
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45975
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45063/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/testchainca.cert with env {}
W20260927 05:13:15.589557 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:15.589818 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:15.590009 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:15.596840 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:15.596964 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:15.597057 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:15.597162 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:15.597237 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:13:15.597327 5648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:13:15.597386 5648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45063/jwks.json
I20260927 05:13:15.606586 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45063/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:15.607746 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:15.608954 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:15.620153 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:15.620100 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:15.620466 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:15.621091 5648 server_base.cc:1034] running on GCE node
I20260927 05:13:15.621769 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:15.623086 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:15.624300 5648 hybrid_clock.cc:648] HybridClock initialized: now 1790485995624268 us; error 38 us; skew 500 ppm
I20260927 05:13:15.624635 5648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:15.626670 5648 webserver.cc:460] Webserver started at http://127.0.97.62:44075/ using document root <none> and password file <none>
I20260927 05:13:15.627394 5648 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:15.627530 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:15.627837 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:15.630434 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "52dafa9d07254a72bc573184a468a08e"
format_stamp: "Formatted at 2026-09-27 05:13:15 on dist-test-slave-1t6p"
server_key: "c10fe583aee71ff53227a008eb047f67"
server_key_iv: "9f2cce6a1cb59b8c9b7f344b8d68a22e"
server_key_version: "encryptionkey@0"
I20260927 05:13:15.631253 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "52dafa9d07254a72bc573184a468a08e"
format_stamp: "Formatted at 2026-09-27 05:13:15 on dist-test-slave-1t6p"
server_key: "c10fe583aee71ff53227a008eb047f67"
server_key_iv: "9f2cce6a1cb59b8c9b7f344b8d68a22e"
server_key_version: "encryptionkey@0"
I20260927 05:13:15.635826 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:15.638947 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:15.640185 5648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:13:15.640373 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "52dafa9d07254a72bc573184a468a08e"
format_stamp: "Formatted at 2026-09-27 05:13:15 on dist-test-slave-1t6p"
server_key: "c10fe583aee71ff53227a008eb047f67"
server_key_iv: "9f2cce6a1cb59b8c9b7f344b8d68a22e"
server_key_version: "encryptionkey@0"
I20260927 05:13:15.640578 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:15.659113 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:15.660070 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:15.660377 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:15.679745 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45975
I20260927 05:13:15.679776 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45975 every 8 connection(s)
I20260927 05:13:15.680996 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:15.683185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5648
I20260927 05:13:15.683389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:15.683987 388 external_mini_cluster.cc:1428] Setting key eb25cfa984cd35df180d8a22c12e554d
I20260927 05:13:15.684846 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:15.696650 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:15.685642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.97.62:45975)
0927 05:13:15.686490 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:15.686500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:15.686541 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:13:15.688098 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:15.688119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:15.688573 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:15.688898 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:15.689451 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.690758 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.692540 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.693149 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.693269 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:15.694455 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:15.694462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:15.694474 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:15.694985 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:15.695130 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:15.695150 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:15.695249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:15.695461 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 05:13:15.695783 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":307,"threads_started":1}
I20260927 05:13:15.697675 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e: Bootstrap starting.
I20260927 05:13:15.700348 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:13:15.700928 5714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:15.702049 5714 log.cc:824] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:15.704805 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e: No bootstrap required, opened a new log
I20260927 05:13:15.711649 5714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260927 05:13:15.711990 5714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:15.712100 5714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dafa9d07254a72bc573184a468a08e, State: Initialized, Role: FOLLOWER
I20260927 05:13:15.712602 5714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260927 05:13:15.712826 5714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:15.712960 5714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:15.713132 5714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:15.715157 5714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260927 05:13:15.715622 5714 leader_election.cc:302] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52dafa9d07254a72bc573184a468a08e; no voters:
I20260927 05:13:15.716231 5714 leader_election.cc:288] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:15.716337 5719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:15.717556 5719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [term 1 LEADER]: Becoming Leader. State: Replica: 52dafa9d07254a72bc573184a468a08e, State: Running, Role: LEADER
I20260927 05:13:15.718072 5714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:15.718020 5719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260927 05:13:15.720638 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52dafa9d07254a72bc573184a468a08e. Latest consensus state: current_term: 1 leader_uuid: "52dafa9d07254a72bc573184a468a08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } } }
I20260927 05:13:15.720623 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52dafa9d07254a72bc573184a468a08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dafa9d07254a72bc573184a468a08e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } } }
I20260927 05:13:15.721060 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:15.721060 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:15.727284 5726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:15.732715 5726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:15.741048 5726 catalog_manager.cc:1347] Generated new cluster ID: c4c1378cbf2d4d37969372e31405f7f1
I20260927 05:13:15.741153 5726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:15.752292 5726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:15.753860 5726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:15.772676 5726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52dafa9d07254a72bc573184a468a08e: Generated new TSK 0
I20260927 05:13:15.773615 5726 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 05:13:15.808511 5744 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 05:13:15.808650 5744 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:13:15.809741 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:15.789844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.97.62:45975)
0927 05:13:15.790066 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:15.790073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:15.790439 (+ 366us) server_negotiation.cc:407] Connection header received
0927 05:13:15.790627 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:15.790634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:15.790717 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:15.790876 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:13:15.791384 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.792502 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.793762 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.794451 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.794578 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:15.795274 (+ 696us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:13:15.809079 (+ 13805us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 05:13:15.809435 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:13:15.809754 5743 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:15.789715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45975 (local address 127.0.0.1:46172)
0927 05:13:15.790339 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:15.790355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:15.790466 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:15.790900 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:15.790904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:15.790913 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:13:15.791212 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:15.791221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.792642 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.792646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:15.793629 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:15.793639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:15.794580 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:15.794615 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:15.794749 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:15.794778 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:13:15.809278 (+ 14500us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 05:13:15.809477 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45975: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":372,"threads_started":1}
W20260927 05:13:15.809862 5717 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 05:13:15.811605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5648
I20260927 05:13:15.812723 5709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:15.902173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5648
2026-09-27T05:13:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 05:13:15.955890 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:13:15.957861 388 webserver.cc:460] Webserver started at https://127.0.0.1:39171/ using document root <none> and password file <none>
I20260927 05:13:15.958097 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:13:15.958726 388 webserver.cc:460] Webserver started at http://127.0.0.1:34809/ using document root <none> and password file <none>
2026-09-27T05:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:15Z Disabled control of system clock
I20260927 05:13:15.983177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33481
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 05:13:16.285296 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:16.285554 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:16.285633 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:16.292527 5751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:16.292611 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:16.292647 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:16.292702 5751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:16.292737 5751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:13:16.292795 5751 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:13:16.292834 5751 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39171/jwks.json
I20260927 05:13:16.301378 5751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:16.302488 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:16.304101 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:16.315658 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:16.315742 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:16.315717 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:16.316594 5751 server_base.cc:1034] running on GCE node
I20260927 05:13:16.317281 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:16.318539 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:16.319756 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790485996319722 us; error 45 us; skew 500 ppm
I20260927 05:13:16.320073 5751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:16.322111 5751 webserver.cc:460] Webserver started at http://127.0.97.62:37507/ using document root <none> and password file <none>
I20260927 05:13:16.322729 5751 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:16.322855 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:16.323231 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:16.325803 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "59e8dd36b49e43b6879ebc83ac726889"
format_stamp: "Formatted at 2026-09-27 05:13:16 on dist-test-slave-1t6p"
server_key: "a8c1d87cb54e2e7880449b2e26979631"
server_key_iv: "f639217474df33327d3807f4fd1ec2a1"
server_key_version: "encryptionkey@0"
I20260927 05:13:16.326565 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "59e8dd36b49e43b6879ebc83ac726889"
format_stamp: "Formatted at 2026-09-27 05:13:16 on dist-test-slave-1t6p"
server_key: "a8c1d87cb54e2e7880449b2e26979631"
server_key_iv: "f639217474df33327d3807f4fd1ec2a1"
server_key_version: "encryptionkey@0"
I20260927 05:13:16.331240 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260927 05:13:16.334492 5765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:16.335827 5751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:16.336038 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "59e8dd36b49e43b6879ebc83ac726889"
format_stamp: "Formatted at 2026-09-27 05:13:16 on dist-test-slave-1t6p"
server_key: "a8c1d87cb54e2e7880449b2e26979631"
server_key_iv: "f639217474df33327d3807f4fd1ec2a1"
server_key_version: "encryptionkey@0"
I20260927 05:13:16.336252 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:16.364002 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:16.364770 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:16.365042 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:16.382176 5751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33481
I20260927 05:13:16.382195 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33481 every 8 connection(s)
I20260927 05:13:16.383394 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:16.386826 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:16.388731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5751
I20260927 05:13:16.388918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:16.389706 388 external_mini_cluster.cc:1428] Setting key 82ebf2569f640452aa6eb1040cbdbc1b
I20260927 05:13:16.398780 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889: Bootstrap starting.
I20260927 05:13:16.402184 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:16.402535 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:16.391554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.0.97.62:33481)
0927 05:13:16.392547 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:16.392557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:16.392597 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:13:16.394110 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:16.394130 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:16.394534 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:16.394824 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:16.395399 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.396674 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.398410 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.399151 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.399272 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:16.400437 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:16.400443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:16.400456 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:16.400928 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:16.401022 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:16.401041 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:16.401121 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:16.401359 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 05:13:16.401663 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":383,"threads_started":1}
I20260927 05:13:16.403304 5817 log.cc:824] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:16.405961 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889: No bootstrap required, opened a new log
I20260927 05:13:16.406217 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:13:16.413717 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 05:13:16.413993 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:16.414062 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e8dd36b49e43b6879ebc83ac726889, State: Initialized, Role: FOLLOWER
I20260927 05:13:16.414584 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 05:13:16.414770 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:16.414861 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:16.414974 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:16.417155 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 05:13:16.417572 5817 leader_election.cc:302] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e8dd36b49e43b6879ebc83ac726889; no voters:
I20260927 05:13:16.418040 5817 leader_election.cc:288] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:16.418171 5822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:16.418846 5822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [term 1 LEADER]: Becoming Leader. State: Replica: 59e8dd36b49e43b6879ebc83ac726889, State: Running, Role: LEADER
I20260927 05:13:16.419389 5822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 05:13:16.419919 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:16.421962 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e8dd36b49e43b6879ebc83ac726889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260927 05:13:16.422289 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:16.423249 5824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e8dd36b49e43b6879ebc83ac726889. Latest consensus state: current_term: 1 leader_uuid: "59e8dd36b49e43b6879ebc83ac726889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8dd36b49e43b6879ebc83ac726889" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260927 05:13:16.423621 5824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:16.427608 5828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:16.433705 5828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:16.441196 5828 catalog_manager.cc:1347] Generated new cluster ID: b8c24e5b99ea4c78b6e849c29c9a44f8
I20260927 05:13:16.441286 5828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:16.452862 5828 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:16.454607 5828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:16.460978 5828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e8dd36b49e43b6879ebc83ac726889: Generated new TSK 0
I20260927 05:13:16.461851 5828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:13:16.493010 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:16.484871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.0.97.62:33481)
0927 05:13:16.485063 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:16.485069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:16.485453 (+ 384us) server_negotiation.cc:407] Connection header received
0927 05:13:16.485647 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:16.485652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:16.485737 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:16.485896 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:13:16.486310 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.487465 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.488804 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.489496 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.489627 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:16.490059 (+ 432us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:13:16.492385 (+ 2326us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39171/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 05:13:16.492695 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39171/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 05:13:16.493129 5820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39171/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 05:13:16.494673 5846 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:16.484727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33481 (local address 127.0.0.1:51664)
0927 05:13:16.485357 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:16.485371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:16.485482 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:16.485910 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:16.485914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:16.485922 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:13:16.486181 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:16.486189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.487640 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.487644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:16.488667 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:16.488678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:16.489611 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:16.489666 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:16.489786 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:16.489815 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:13:16.492590 (+ 2775us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39171/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 05:13:16.494253 (+ 1663us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39171/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1393,"thread_start_us":361,"threads_started":1}
I20260927 05:13:16.495687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5751
I20260927 05:13:16.497359 5812 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 05:13:16.508867 5847 webserver.cc:533] Webserver: sslize failed:
E20260927 05:13:16.509001 5847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:13:16.587888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5751
2026-09-27T05:13:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (687 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T05:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:16Z Disabled control of system clock
I20260927 05:13:16.666421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33779
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33779
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/testchainca.cert with env {}
W20260927 05:13:16.972133 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:16.972399 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:16.972479 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:16.979264 5852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:16.979350 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:16.979387 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:16.979442 5852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:16.979478 5852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:13:16.979537 5852 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:13:16.979599 5852 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 05:13:16.988713 5852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:16.989943 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:16.991159 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:17.002211 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:17.002413 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:17.002254 5860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:17.003306 5852 server_base.cc:1034] running on GCE node
I20260927 05:13:17.003997 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:17.005250 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:17.006469 5852 hybrid_clock.cc:648] HybridClock initialized: now 1790485997006433 us; error 53 us; skew 500 ppm
I20260927 05:13:17.006785 5852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:13:17.009572 5852 webserver.cc:460] Webserver started at http://127.0.97.62:45295/ using document root <none> and password file <none>
I20260927 05:13:17.010391 5852 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:17.010530 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:17.010926 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:17.014003 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "533830b968ee409abb0d0e0d427cf396"
format_stamp: "Formatted at 2026-09-27 05:13:17 on dist-test-slave-1t6p"
server_key: "81d20a2906dbdc6531f40af0f9647e3b"
server_key_iv: "b89f27d4efaeb01a73eddf6aaed4d372"
server_key_version: "encryptionkey@0"
I20260927 05:13:17.014775 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "533830b968ee409abb0d0e0d427cf396"
format_stamp: "Formatted at 2026-09-27 05:13:17 on dist-test-slave-1t6p"
server_key: "81d20a2906dbdc6531f40af0f9647e3b"
server_key_iv: "b89f27d4efaeb01a73eddf6aaed4d372"
server_key_version: "encryptionkey@0"
I20260927 05:13:17.019450 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:13:17.022639 5866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:17.023871 5852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:17.024055 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "533830b968ee409abb0d0e0d427cf396"
format_stamp: "Formatted at 2026-09-27 05:13:17 on dist-test-slave-1t6p"
server_key: "81d20a2906dbdc6531f40af0f9647e3b"
server_key_iv: "b89f27d4efaeb01a73eddf6aaed4d372"
server_key_version: "encryptionkey@0"
I20260927 05:13:17.024264 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:17.047593 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:17.048326 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:17.048609 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:17.066107 5852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33779
I20260927 05:13:17.066128 5917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33779 every 8 connection(s)
I20260927 05:13:17.067313 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:17.070664 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:17.071403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5852
I20260927 05:13:17.071626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:17.072413 388 external_mini_cluster.cc:1428] Setting key abf820032cf1f64f1bde20dad34e5411
I20260927 05:13:17.082466 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396: Bootstrap starting.
I20260927 05:13:17.084792 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:17.074215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.0.97.62:33779)
0927 05:13:17.075195 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:17.075206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:17.075245 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:13:17.076816 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:17.076835 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:17.077253 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:17.077605 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:17.078130 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.079418 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.080668 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.081312 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.081490 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:17.082638 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:17.082646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:17.082659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:13:17.083237 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:17.083310 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:13:17.083329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:17.083447 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:17.083673 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 05:13:17.083948 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":337,"threads_started":1}
I20260927 05:13:17.085541 5918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:17.086752 5918 log.cc:824] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:17.088511 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:13:17.089463 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396: No bootstrap required, opened a new log
I20260927 05:13:17.096798 5918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260927 05:13:17.097123 5918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:17.097227 5918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533830b968ee409abb0d0e0d427cf396, State: Initialized, Role: FOLLOWER
I20260927 05:13:17.097829 5918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260927 05:13:17.098067 5918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:17.098196 5918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:17.098361 5918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:17.100689 5918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260927 05:13:17.101145 5918 leader_election.cc:302] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 533830b968ee409abb0d0e0d427cf396; no voters:
I20260927 05:13:17.101665 5918 leader_election.cc:288] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:17.101788 5923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:17.102440 5923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [term 1 LEADER]: Becoming Leader. State: Replica: 533830b968ee409abb0d0e0d427cf396, State: Running, Role: LEADER
I20260927 05:13:17.103057 5923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260927 05:13:17.103704 5918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:17.105679 5924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "533830b968ee409abb0d0e0d427cf396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } } }
I20260927 05:13:17.106117 5924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:17.106105 5925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533830b968ee409abb0d0e0d427cf396. Latest consensus state: current_term: 1 leader_uuid: "533830b968ee409abb0d0e0d427cf396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533830b968ee409abb0d0e0d427cf396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } } }
I20260927 05:13:17.106443 5925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:17.107350 5931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:17.112617 5931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:17.120827 5931 catalog_manager.cc:1347] Generated new cluster ID: 52d964d494884c7c9a54ab954c1d642e
I20260927 05:13:17.120924 5931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:17.136485 5931 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:17.137557 5931 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:17.146189 5931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533830b968ee409abb0d0e0d427cf396: Generated new TSK 0
I20260927 05:13:17.147145 5931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:13:17.238528 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:17.170614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.97.62:33779)
0927 05:13:17.170791 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:17.170803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:17.171185 (+ 382us) server_negotiation.cc:407] Connection header received
0927 05:13:17.171462 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:17.171466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:17.171537 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:17.171683 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:13:17.172121 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.172826 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.174158 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.174606 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.174734 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:17.175212 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:13:17.237769 (+ 62557us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 05:13:17.238226 (+ 457us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49498: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 05:13:17.238670 5921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49498: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 05:13:17.238818 5947 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:17.170502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33779 (local address 127.0.0.1:49498)
0927 05:13:17.171094 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:17.171109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:17.171261 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:17.171696 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:17.171701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:17.171713 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:13:17.171980 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:17.171989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.172981 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.172986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:17.174019 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:17.174029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:17.174727 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:17.174780 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:17.174920 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:17.174956 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:13:17.238102 (+ 63146us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 05:13:17.238388 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":286,"threads_started":1}
I20260927 05:13:17.240302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5852
I20260927 05:13:17.241473 5913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:17.336493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5852
2026-09-27T05:13:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (726 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (393 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1452 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:19 dist-test-slave-1t6p krb5kdc[5963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:19 dist-test-slave-1t6p krb5kdc[5963](info): set up 2 sockets
Sep 27 05:13:19 dist-test-slave-1t6p krb5kdc[5963](info): commencing operation
krb5kdc: starting...
W20260927 05:13:20.259792 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:20 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:20.443292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40901
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:20.751652 5976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:20.751948 5976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:20.752115 5976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:20.758781 5976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:20.758914 5976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:20.759040 5976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:20.759145 5976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:20.759233 5976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:20.768559 5976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:20.769753 5976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:20.771054 5976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:20.782114 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:20.782183 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:20.782152 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:20.783303 5976 server_base.cc:1034] running on GCE node
I20260927 05:13:20.783987 5976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:20.785194 5976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:20.786412 5976 hybrid_clock.cc:648] HybridClock initialized: now 1790486000786368 us; error 52 us; skew 500 ppm
I20260927 05:13:20.786751 5976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:20 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:20.794674 5976 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:20.796156 5976 webserver.cc:460] Webserver started at http://127.0.97.62:39331/ using document root <none> and password file <none>
I20260927 05:13:20.796825 5976 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:20.796967 5976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:20.797266 5976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:20.800068 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "ba9200e460f0462ea73219923d3147ec"
format_stamp: "Formatted at 2026-09-27 05:13:20 on dist-test-slave-1t6p"
server_key: "f8de9a4459b4512b982d3c118c51ef3d"
server_key_iv: "c868ac532a6a91d37052011ddb3d4559"
server_key_version: "encryptionkey@0"
I20260927 05:13:20.800794 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba9200e460f0462ea73219923d3147ec"
format_stamp: "Formatted at 2026-09-27 05:13:20 on dist-test-slave-1t6p"
server_key: "f8de9a4459b4512b982d3c118c51ef3d"
server_key_iv: "c868ac532a6a91d37052011ddb3d4559"
server_key_version: "encryptionkey@0"
I20260927 05:13:20.805572 5976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:20.808717 5991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:20.809834 5976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:20.810009 5976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "ba9200e460f0462ea73219923d3147ec"
format_stamp: "Formatted at 2026-09-27 05:13:20 on dist-test-slave-1t6p"
server_key: "f8de9a4459b4512b982d3c118c51ef3d"
server_key_iv: "c868ac532a6a91d37052011ddb3d4559"
server_key_version: "encryptionkey@0"
I20260927 05:13:20.810191 5976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:20.821558 5976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:20.825127 5976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:20.825389 5976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:20.842955 5976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40901
I20260927 05:13:20.842978 6042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40901 every 8 connection(s)
I20260927 05:13:20.844269 5976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:20.847904 6043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:20.849334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 5976
I20260927 05:13:20.849574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:20.850338 388 external_mini_cluster.cc:1428] Setting key d2f4b06e739e7b01b207163ba67bc517
I20260927 05:13:20.861753 6043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec: Bootstrap starting.
Sep 27 05:13:20 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:20.865270 6043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:20.866302 6043 log.cc:824] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:20.869340 6043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec: No bootstrap required, opened a new log
I20260927 05:13:20.872897 6046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:20.852238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.97.62:40901)
0927 05:13:20.853003 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:20.853014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:20.853050 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:13:20.854527 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:20.854550 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:20.854917 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:20.855292 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:20.856938 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:20.858155 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:20.859934 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:20.860566 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:20.860684 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:20.865139 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:20.865145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:20.865159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:20.865194 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:20.868706 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:20.869296 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:20.869306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:20.869317 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:20.869508 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:20.869923 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:20.869928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:20.869931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:20.870569 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:20.870840 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:20.871205 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 05:13:20.871631 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
I20260927 05:13:20.878106 6043 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } }
I20260927 05:13:20.878391 6043 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:20.878456 6043 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9200e460f0462ea73219923d3147ec, State: Initialized, Role: FOLLOWER
I20260927 05:13:20.878912 6043 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } }
I20260927 05:13:20.879156 6043 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:20.879252 6043 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:20.879364 6043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:20.881469 6043 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } }
I20260927 05:13:20.881935 6043 leader_election.cc:302] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9200e460f0462ea73219923d3147ec; no voters:
I20260927 05:13:20.882485 6043 leader_election.cc:288] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:20.882638 6048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:20.883309 6048 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [term 1 LEADER]: Becoming Leader. State: Replica: ba9200e460f0462ea73219923d3147ec, State: Running, Role: LEADER
I20260927 05:13:20.883829 6048 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } }
I20260927 05:13:20.884547 6043 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:20.886793 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9200e460f0462ea73219923d3147ec. Latest consensus state: current_term: 1 leader_uuid: "ba9200e460f0462ea73219923d3147ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } } }
I20260927 05:13:20.886767 6049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba9200e460f0462ea73219923d3147ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9200e460f0462ea73219923d3147ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40901 } } }
I20260927 05:13:20.887346 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:20.887347 6049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:20.892133 6054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:20.896988 6054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:20.904523 6054 catalog_manager.cc:1347] Generated new cluster ID: e0975039bc544dcbadfcbed6b7459af8
I20260927 05:13:20.904625 6054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:20.924676 6054 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:20.926447 6054 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:20.934029 6054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba9200e460f0462ea73219923d3147ec: Generated new TSK 0
I20260927 05:13:20.934935 6054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:13:20.987594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:21.299796 6069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:21.300097 6069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:21.300261 6069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:21.307301 6069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:21.307441 6069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:21.307608 6069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:21.317126 6069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:21.318264 6069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:21.319585 6069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:21.331997 6074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:21.331943 6075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:21.332037 6077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:21.332942 6069 server_base.cc:1034] running on GCE node
I20260927 05:13:21.333455 6069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:21.334287 6069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:21.335501 6069 hybrid_clock.cc:648] HybridClock initialized: now 1790486001335467 us; error 47 us; skew 500 ppm
I20260927 05:13:21.335788 6069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:21 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:21.338764 6069 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:13:21.340161 6069 webserver.cc:460] Webserver started at http://127.0.97.1:43809/ using document root <none> and password file <none>
I20260927 05:13:21.340746 6069 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:21.340842 6069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:21.341075 6069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:21.343842 6069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "4ace963d9b0c40e8b1d1e088e328d362"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "6ef99c1b8f82b1f1df96febee8b978d7"
server_key_iv: "b3c22334384d9fb43405e4503424b0cd"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.344523 6069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4ace963d9b0c40e8b1d1e088e328d362"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "6ef99c1b8f82b1f1df96febee8b978d7"
server_key_iv: "b3c22334384d9fb43405e4503424b0cd"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.349251 6069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:21.352311 6084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.353484 6069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:21.353628 6069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "4ace963d9b0c40e8b1d1e088e328d362"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "6ef99c1b8f82b1f1df96febee8b978d7"
server_key_iv: "b3c22334384d9fb43405e4503424b0cd"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.353806 6069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:21.393977 6069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:21.398236 6069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:21.398535 6069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:21.399510 6069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:21.400789 6069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:21.400859 6069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.401039 6069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:21.401103 6069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.432942 6069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37649
I20260927 05:13:21.433024 6196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37649 every 8 connection(s)
I20260927 05:13:21.434301 6069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:21.436874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6069
I20260927 05:13:21.437134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:21.438112 388 external_mini_cluster.cc:1428] Setting key 44d3b631a5a89bdbf5bcd494c29352fd
Sep 27 05:13:21 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:21.456740 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:21.436872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53253 (local address 127.0.97.62:40901)
0927 05:13:21.440630 (+ 3758us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:21.440634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:21.440648 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:13:21.440688 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:21.440691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:21.440739 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:21.440820 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:21.442863 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.443618 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.446127 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.446524 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.446626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:21.449952 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:21.449961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:21.449969 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:21.450012 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:21.453524 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:21.454185 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:21.454193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:21.454211 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:21.454351 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:21.454761 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:21.454769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:21.454775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:21.455175 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:21.455335 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:21.456005 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 05:13:21.456263 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3668,"thread_start_us":3585,"threads_started":1}
I20260927 05:13:21.457454 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:21.437376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40901 (local address 127.0.97.1:53253)
0927 05:13:21.438162 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:21.438205 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:21.439951 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:21.441114 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:21.441125 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:21.441873 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:21.442695 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:21.442722 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.443767 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.443774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:21.445960 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:21.445977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.446760 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.446769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:21.446993 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:21.447705 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:21.447726 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:21.449738 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:21.453657 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:21.453663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:21.453676 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:21.454067 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:21.454463 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:21.454466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:21.454468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:21.454650 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:21.455328 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:21.455336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:21.455632 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 05:13:21.455865 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 05:13:21.456232 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
I20260927 05:13:21.459095 6197 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40901
I20260927 05:13:21.459393 6197 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:13:21.460091 6197 heartbeater.cc:511] Master 127.0.97.62:40901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:13:21.462361 6008 ts_manager.cc:194] Registered new tserver with Master: 4ace963d9b0c40e8b1d1e088e328d362 (127.0.97.1:37649)
I20260927 05:13:21.463820 6008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:13:21.484385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:21.800170 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:21.800413 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:21.800501 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:21.807317 6203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:21.807404 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:21.807514 6203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:21.816959 6203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:21.818102 6203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:21.819317 6203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:21.831787 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:21.831812 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:21.831851 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:21.832670 6203 server_base.cc:1034] running on GCE node
I20260927 05:13:21.833312 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:21.834208 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:21.835395 6203 hybrid_clock.cc:648] HybridClock initialized: now 1790486001835349 us; error 35 us; skew 500 ppm
I20260927 05:13:21.835692 6203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:21 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:21.838670 6203 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:13:21.840108 6203 webserver.cc:460] Webserver started at http://127.0.97.2:36535/ using document root <none> and password file <none>
I20260927 05:13:21.840794 6203 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:21.840920 6203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:21.841215 6203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:21.843997 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "85ea9cdd67414ed8b8077cbd1a1e853b"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "2d16d0b8259ca11e6419723148416f4a"
server_key_iv: "71aa1cb3eb5f02afa99f86aa893c50e2"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.844746 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "85ea9cdd67414ed8b8077cbd1a1e853b"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "2d16d0b8259ca11e6419723148416f4a"
server_key_iv: "71aa1cb3eb5f02afa99f86aa893c50e2"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.849536 6203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:21.853008 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.854298 6203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:13:21.854446 6203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "85ea9cdd67414ed8b8077cbd1a1e853b"
format_stamp: "Formatted at 2026-09-27 05:13:21 on dist-test-slave-1t6p"
server_key: "2d16d0b8259ca11e6419723148416f4a"
server_key_iv: "71aa1cb3eb5f02afa99f86aa893c50e2"
server_key_version: "encryptionkey@0"
I20260927 05:13:21.854632 6203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:21.871784 6203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:21.875598 6203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:21.875859 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:21.876698 6203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:21.877942 6203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:21.878022 6203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.878161 6203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:21.878209 6203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:21.911520 6203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40927
I20260927 05:13:21.911625 6330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40927 every 8 connection(s)
I20260927 05:13:21.912797 6203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:21.921743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6203
I20260927 05:13:21.921936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:21.922341 388 external_mini_cluster.cc:1428] Setting key 073cfa920fb68b344e33581b626b4560
Sep 27 05:13:21 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:21.932889 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:21.915440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44903 (local address 127.0.97.62:40901)
0927 05:13:21.915608 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:21.915613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:21.916764 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 05:13:21.918487 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:21.918494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:21.918584 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:21.918692 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:21.920137 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.920839 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.922421 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.922836 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.922965 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:21.926529 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:21.926538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:21.926545 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:21.926594 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:21.930018 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:21.930686 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:21.930693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:21.930696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:21.930823 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:21.931281 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:21.931300 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:21.931303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:21.931565 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:21.931716 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:21.932330 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 05:13:21.932531 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:13:21.933804 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:21.915956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40901 (local address 127.0.97.2:44903)
0927 05:13:21.916619 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:21.916661 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:21.918208 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:21.918915 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:21.918923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:21.919455 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:21.919965 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:21.919990 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.920997 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.921001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:21.922281 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:21.922292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:21.923082 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:21.923090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:21.923377 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:21.924050 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:21.924075 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:21.926306 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:21.930154 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:21.930160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:21.930172 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:21.930558 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:21.930931 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:21.930937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:21.930941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:21.931174 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:21.931688 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:21.931696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:21.931998 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 05:13:21.932242 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 05:13:21.932736 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":267,"threads_started":1}
I20260927 05:13:21.935210 6331 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40901
I20260927 05:13:21.935501 6331 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:21.936331 6331 heartbeater.cc:511] Master 127.0.97.62:40901 requested a full tablet report, sending...
I20260927 05:13:21.937752 6008 ts_manager.cc:194] Registered new tserver with Master: 85ea9cdd67414ed8b8077cbd1a1e853b (127.0.97.2:40927)
I20260927 05:13:21.938503 6008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44903
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:13:21.967941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:22.278831 6336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:22.279170 6336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:22.279333 6336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:22.286448 6336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:22.286567 6336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:22.286738 6336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:22.296468 6336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:22.297580 6336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:22.298786 6336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:22.311303 6344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:22.311368 6342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:22.311554 6341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:22.311903 6336 server_base.cc:1034] running on GCE node
I20260927 05:13:22.312578 6336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:22.313527 6336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:22.314750 6336 hybrid_clock.cc:648] HybridClock initialized: now 1790486002314715 us; error 53 us; skew 500 ppm
I20260927 05:13:22.315131 6336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.318172 6336 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:13:22.319499 6336 webserver.cc:460] Webserver started at http://127.0.97.3:33139/ using document root <none> and password file <none>
I20260927 05:13:22.320127 6336 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:22.320264 6336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:22.320537 6336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:22.323256 6336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "b996b1cece794384b199e256ef0eaa2f"
format_stamp: "Formatted at 2026-09-27 05:13:22 on dist-test-slave-1t6p"
server_key: "e6157cf7c2329fa4316ebe5e231fcad4"
server_key_iv: "ac2577eb5795f01c81763297badc3788"
server_key_version: "encryptionkey@0"
I20260927 05:13:22.324002 6336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b996b1cece794384b199e256ef0eaa2f"
format_stamp: "Formatted at 2026-09-27 05:13:22 on dist-test-slave-1t6p"
server_key: "e6157cf7c2329fa4316ebe5e231fcad4"
server_key_iv: "ac2577eb5795f01c81763297badc3788"
server_key_version: "encryptionkey@0"
I20260927 05:13:22.328601 6336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:22.331665 6351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:22.332875 6336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:13:22.333037 6336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "b996b1cece794384b199e256ef0eaa2f"
format_stamp: "Formatted at 2026-09-27 05:13:22 on dist-test-slave-1t6p"
server_key: "e6157cf7c2329fa4316ebe5e231fcad4"
server_key_iv: "ac2577eb5795f01c81763297badc3788"
server_key_version: "encryptionkey@0"
I20260927 05:13:22.333250 6336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:22.350457 6336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:22.354424 6336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:22.354699 6336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:22.355562 6336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:22.356789 6336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:22.356894 6336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:22.357048 6336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:22.357125 6336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:22.389423 6336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40069
I20260927 05:13:22.389465 6463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40069 every 8 connection(s)
I20260927 05:13:22.390653 6336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:22.394733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6336
I20260927 05:13:22.394920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:22.395345 388 external_mini_cluster.cc:1428] Setting key cc3f56dde818b58e1b4494740935e0fe
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:22.409422 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.392998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33889 (local address 127.0.97.62:40901)
0927 05:13:22.393162 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.393167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.394304 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 05:13:22.396224 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.396229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.396311 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.396430 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.397842 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.398668 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.400242 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.400639 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.400784 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.403939 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.403945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.403950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.403987 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.406512 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.407273 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.407280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.407288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.407393 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.407797 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.407802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.407805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.408041 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.408206 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.408869 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.409077 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:22.410315 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.393479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40901 (local address 127.0.97.3:33889)
0927 05:13:22.394151 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:22.394187 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:22.395943 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:22.396672 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:22.396679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:22.397167 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:22.397681 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:22.397696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.398813 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.398825 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:22.400100 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:22.400112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.400866 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.400875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:22.401099 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.401734 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:22.401757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:22.403767 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:22.406648 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:22.406657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:22.406670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:22.407156 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:22.407500 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:22.407506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:22.407510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:22.407686 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:22.408162 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:22.408169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:22.408480 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 05:13:22.408719 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 05:13:22.409173 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":266,"threads_started":1}
I20260927 05:13:22.411468 6464 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40901
I20260927 05:13:22.411716 6464 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:22.412318 6464 heartbeater.cc:511] Master 127.0.97.62:40901 requested a full tablet report, sending...
I20260927 05:13:22.413594 6008 ts_manager.cc:194] Registered new tserver with Master: b996b1cece794384b199e256ef0eaa2f (127.0.97.3:40069)
I20260927 05:13:22.414427 6008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33889
I20260927 05:13:22.421659 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/test-admin.keytab.
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.446141 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 05:13:22.450655 6471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:22.448496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.97.1:37649)
0927 05:13:22.449106 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.449113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.449136 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:13:22.449187 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.449192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.449367 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.449479 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.450214 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45958: BlockingRecv error: recv got EOF from 127.0.0.1:45958 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":331,"threads_started":1}
W20260927 05:13:22.450838 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.452469 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.453727 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:22.467214 6197 heartbeater.cc:503] Master 127.0.97.62:40901 was elected leader, sending a full tablet report...
I20260927 05:13:22.468374 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.455206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.97.1:37649)
0927 05:13:22.455403 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.455409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.455583 (+ 174us) server_negotiation.cc:407] Connection header received
0927 05:13:22.455763 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.455769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.455834 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.455932 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.456864 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.457882 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.459326 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.459839 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.459973 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.463199 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.463210 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.463226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.463259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.465679 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.466336 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.466344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.466348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.466465 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.466841 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.466846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.466849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.467277 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.467570 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.467890 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.468130 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 05:13:22.473596 6471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:22.471969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.0.97.1:37649)
0927 05:13:22.472160 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.472166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.472470 (+ 304us) server_negotiation.cc:407] Connection header received
0927 05:13:22.472619 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.472623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.472678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.472750 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.473342 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45978: BlockingRecv error: recv got EOF from 127.0.0.1:45978 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 05:13:22.473764 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.475211 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.476507 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:22.489404 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.477586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.0.97.1:37649)
0927 05:13:22.477712 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.477716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.478101 (+ 385us) server_negotiation.cc:407] Connection header received
0927 05:13:22.478192 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.478195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.478241 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.478311 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.479217 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.479843 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.480936 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.481354 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.481440 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.484840 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.484844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.484847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.484873 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.487333 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.487899 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.487903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.487905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.488009 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.488304 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.488307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.488309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.488528 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.488696 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.488935 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.489121 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 05:13:22.493217 6471 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:22.491695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.0.97.1:37649)
0927 05:13:22.491873 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.491878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.492105 (+ 227us) server_negotiation.cc:407] Connection header received
0927 05:13:22.492178 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.492181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.492243 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.492321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.493026 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46002: BlockingRecv error: recv got EOF from 127.0.0.1:46002 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 05:13:22.493357 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485951045978-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.494747 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:22.496001 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:22.509415 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.497277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.97.1:37649)
0927 05:13:22.497461 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.497466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.497617 (+ 151us) server_negotiation.cc:407] Connection header received
0927 05:13:22.497807 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.497817 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.497867 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.497960 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.498818 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.499557 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.500767 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.501163 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.501291 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.504595 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.504600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.504603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.504630 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.507079 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.507642 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.507649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.507653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.507801 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.508155 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.508161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.508164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.508418 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.508740 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.508928 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.509121 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:13:22.510849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6069
I20260927 05:13:22.521626 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.511594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.97.1:37649)
0927 05:13:22.511725 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.511729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.512067 (+ 338us) server_negotiation.cc:407] Connection header received
0927 05:13:22.512168 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.512171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.512218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.512288 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.513135 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.513803 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.514940 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.515386 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.515479 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.517195 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.517199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.517202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.517227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.519581 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.520124 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.520127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.520129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.520235 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.520525 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.520528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.520530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.520742 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.520908 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.521152 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.521320 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:13:22.522110 6192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:22.603530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6069
I20260927 05:13:22.619726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6203
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:13:22.633996 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.620643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.0.97.2:40927)
0927 05:13:22.621185 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.621191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.621216 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:22.621278 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.621283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.621487 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.621613 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.622466 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.623552 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.624671 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.625201 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.625300 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.628901 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.628910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.628924 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.628953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.631720 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.632287 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.632293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.632295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.632401 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.632683 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.632686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.632687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.632970 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.633188 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.633450 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.633691 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
I20260927 05:13:22.635241 6326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:22.715976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6203
I20260927 05:13:22.731242 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6336
Sep 27 05:13:22 dist-test-slave-1t6p krb5kdc[5963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:13:22.746104 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:22.732145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.0.97.3:40069)
0927 05:13:22.732743 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:22.732750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:22.732778 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:13:22.732838 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:22.732844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:22.733039 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:22.733211 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:22.734237 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.735272 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.736464 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:22.737013 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:22.737151 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:22.740941 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:22.740951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:22.740964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:22.740996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:22.743492 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.744141 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.744148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.744150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.744262 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:22.744627 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:22.744634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:22.744637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:22.744951 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:22.745255 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:22.745546 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 05:13:22.745797 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":323,"threads_started":1}
I20260927 05:13:22.747426 6459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:22.828393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6336
I20260927 05:13:22.844240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 5976
I20260927 05:13:22.845404 6038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:22.943964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 5976
2026-09-27T05:13:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3767 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:23 dist-test-slave-1t6p krb5kdc[6491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:23 dist-test-slave-1t6p krb5kdc[6491](info): set up 2 sockets
Sep 27 05:13:23 dist-test-slave-1t6p krb5kdc[6491](info): commencing operation
krb5kdc: starting...
W20260927 05:13:24.019670 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:24 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 05:13:24.208510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:24.511615 6504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:24.511901 6504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:24.512070 6504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:24.519120 6504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:24.519217 6504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:24.519310 6504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:24.519388 6504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:24.519445 6504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:24.529030 6504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:24.530241 6504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:24.531548 6504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:24.542891 6509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:24.542836 6512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:24.542869 6510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:24.543630 6504 server_base.cc:1034] running on GCE node
I20260927 05:13:24.544341 6504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:24.545552 6504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:24.546748 6504 hybrid_clock.cc:648] HybridClock initialized: now 1790486004546699 us; error 51 us; skew 500 ppm
I20260927 05:13:24.547127 6504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:24 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:24.549978 6504 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 05:13:24.551301 6504 webserver.cc:460] Webserver started at http://127.0.97.62:39121/ using document root <none> and password file <none>
I20260927 05:13:24.551889 6504 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:24.551981 6504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:24.552209 6504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:24.554872 6504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "06b69ab300d2440d8a05e4fac552aab8"
format_stamp: "Formatted at 2026-09-27 05:13:24 on dist-test-slave-1t6p"
server_key: "3e953c5b4448aab0b312b82c9c293977"
server_key_iv: "9f17a63bd8fbbe7873579e135a0303e5"
server_key_version: "encryptionkey@0"
I20260927 05:13:24.555604 6504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "06b69ab300d2440d8a05e4fac552aab8"
format_stamp: "Formatted at 2026-09-27 05:13:24 on dist-test-slave-1t6p"
server_key: "3e953c5b4448aab0b312b82c9c293977"
server_key_iv: "9f17a63bd8fbbe7873579e135a0303e5"
server_key_version: "encryptionkey@0"
I20260927 05:13:24.560277 6504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:24.563781 6519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:24.565392 6504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:13:24.565558 6504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "06b69ab300d2440d8a05e4fac552aab8"
format_stamp: "Formatted at 2026-09-27 05:13:24 on dist-test-slave-1t6p"
server_key: "3e953c5b4448aab0b312b82c9c293977"
server_key_iv: "9f17a63bd8fbbe7873579e135a0303e5"
server_key_version: "encryptionkey@0"
I20260927 05:13:24.565788 6504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:24.582849 6504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:24.586993 6504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:24.587282 6504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:24.604991 6504 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41801
I20260927 05:13:24.605019 6570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41801 every 8 connection(s)
I20260927 05:13:24.606236 6504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:24.609712 6571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:24.615154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6504
I20260927 05:13:24.615368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:24.615993 388 external_mini_cluster.cc:1428] Setting key 14bf16716e62809a99389206b603135d
I20260927 05:13:24.620386 6571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Bootstrap starting.
I20260927 05:13:24.624038 6571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:24.625247 6571 log.cc:824] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:24.628288 6571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: No bootstrap required, opened a new log
Sep 27 05:13:24 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:24.638522 6571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } }
I20260927 05:13:24.638934 6571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:24.639098 6571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b69ab300d2440d8a05e4fac552aab8, State: Initialized, Role: FOLLOWER
I20260927 05:13:24.639721 6571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } }
I20260927 05:13:24.639976 6571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:24.640117 6571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:24.640161 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:24.617807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.0.97.62:41801)
0927 05:13:24.618574 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:24.618589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:24.618635 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:13:24.620888 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:24.620919 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:24.621365 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:24.621772 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:24.622942 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:24.624983 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:24.626725 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:24.627745 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:24.627915 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:24.632279 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:24.632289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:24.632307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:24.632352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:24.636148 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:24.636764 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:24.636777 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:24.636785 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:24.636993 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:24.637317 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:24.637322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:24.637325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:24.637920 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:24.638211 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:24.638465 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 05:13:24.638869 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":314,"threads_started":1}
I20260927 05:13:24.640321 6571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:24.642488 6571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } }
I20260927 05:13:24.642954 6571 leader_election.cc:302] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06b69ab300d2440d8a05e4fac552aab8; no voters:
I20260927 05:13:24.643558 6571 leader_election.cc:288] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:24.643687 6576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:24.645092 6576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 LEADER]: Becoming Leader. State: Replica: 06b69ab300d2440d8a05e4fac552aab8, State: Running, Role: LEADER
I20260927 05:13:24.645504 6571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:24.645552 6576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } }
I20260927 05:13:24.648576 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } }
I20260927 05:13:24.648574 6578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b69ab300d2440d8a05e4fac552aab8. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } }
I20260927 05:13:24.648940 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:24.648994 6578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:24.650442 6583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:24.656472 6583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:24.664454 6583 catalog_manager.cc:1347] Generated new cluster ID: da3c825fa4fe44899b55c976c03a84de
I20260927 05:13:24.664602 6583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:24.703768 6583 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:24.705013 6583 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:24.715586 6583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Generated new TSK 0
I20260927 05:13:24.716364 6583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 05:13:24.769865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:25.083151 6597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:25.083402 6597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:25.083570 6597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:25.090759 6597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:25.090873 6597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:25.091109 6597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:25.100250 6597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:25.101393 6597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:25.102646 6597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:25.115366 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:25.115315 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:25.115398 6605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:25.115841 6597 server_base.cc:1034] running on GCE node
I20260927 05:13:25.116422 6597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:25.117269 6597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:25.118507 6597 hybrid_clock.cc:648] HybridClock initialized: now 1790486005118464 us; error 48 us; skew 500 ppm
I20260927 05:13:25.118811 6597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:25 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:25.121789 6597 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:13:25.123131 6597 webserver.cc:460] Webserver started at http://127.0.97.1:35443/ using document root <none> and password file <none>
I20260927 05:13:25.123759 6597 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:25.123907 6597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:25.124188 6597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:25.126973 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "daa16bb930014639b04b857b99c7e5e0"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "a4b8cb42beec60edcbe7401368218a95"
server_key_iv: "2df4401c14cbf53b07018baa3eefe6bf"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.127766 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "daa16bb930014639b04b857b99c7e5e0"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "a4b8cb42beec60edcbe7401368218a95"
server_key_iv: "2df4401c14cbf53b07018baa3eefe6bf"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.132483 6597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:25.135701 6612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.136868 6597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:13:25.136993 6597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "daa16bb930014639b04b857b99c7e5e0"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "a4b8cb42beec60edcbe7401368218a95"
server_key_iv: "2df4401c14cbf53b07018baa3eefe6bf"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.137137 6597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:25.160296 6597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:25.164075 6597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:25.164295 6597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:25.165138 6597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:25.166337 6597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:25.166407 6597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.166486 6597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:25.166530 6597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.211632 6597 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33065
I20260927 05:13:25.211714 6724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33065 every 8 connection(s)
I20260927 05:13:25.213006 6597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:25.217119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6597
I20260927 05:13:25.217325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:25.218153 388 external_mini_cluster.cc:1428] Setting key 8e92e16894c64ac7e1cd6a39420ba0bf
Sep 27 05:13:25 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:25.236555 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:25.215798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55815 (local address 127.0.97.62:41801)
0927 05:13:25.216261 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:25.216270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:25.217366 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 05:13:25.220414 (+ 3048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:25.220421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:25.220513 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:25.220601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:25.223068 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.224342 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.226313 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.227029 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.227183 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:25.230947 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:25.230953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:25.230959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:25.231047 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:25.233568 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:25.234334 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:25.234341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:25.234344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:25.234467 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:25.234871 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:25.234878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:25.234882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:25.235223 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:25.235369 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:25.235996 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 05:13:25.236201 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260927 05:13:25.237468 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:25.216367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:55815)
0927 05:13:25.217204 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:25.217254 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:25.220083 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:25.221182 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:25.221196 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:25.221988 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:25.222824 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:25.222851 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.224546 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.224553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:25.226165 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:25.226176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.227236 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.227241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:25.227526 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:25.228190 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:25.228223 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:25.230734 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:25.233701 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:25.233710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:25.233725 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:25.234222 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:25.234578 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:25.234584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:25.234587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:25.234758 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:25.235342 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:25.235350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:25.235666 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 05:13:25.235914 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 05:13:25.236390 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:13:25.238948 6725 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:25.239287 6725 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:25.239977 6725 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 05:13:25.242063 6536 ts_manager.cc:194] Registered new tserver with Master: daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:25.243497 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 05:13:25.262414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:25.572959 6731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:25.573212 6731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:25.573302 6731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:25.580322 6731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:25.580416 6731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:25.580523 6731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:25.589947 6731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:25.591243 6731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:25.592454 6731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:25.605103 6736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:25.605093 6737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:25.606084 6731 server_base.cc:1034] running on GCE node
W20260927 05:13:25.607262 6739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:25.607965 6731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:25.609014 6731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:25.610239 6731 hybrid_clock.cc:648] HybridClock initialized: now 1790486005610175 us; error 55 us; skew 500 ppm
I20260927 05:13:25.610584 6731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:25 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:25.614122 6731 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:13:25.616073 6731 webserver.cc:460] Webserver started at http://127.0.97.2:39083/ using document root <none> and password file <none>
I20260927 05:13:25.616727 6731 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:25.616850 6731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:25.617168 6731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:25.619992 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "d42228ac01354c308bd1ff678d80d383"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "e6adb36df0124bc7c075debd161bad61"
server_key_iv: "d53e1cad15bf84ef4561d879cf93a08d"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.620730 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d42228ac01354c308bd1ff678d80d383"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "e6adb36df0124bc7c075debd161bad61"
server_key_iv: "d53e1cad15bf84ef4561d879cf93a08d"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.625407 6731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:13:25.628612 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.629822 6731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:25.629983 6731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "d42228ac01354c308bd1ff678d80d383"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "e6adb36df0124bc7c075debd161bad61"
server_key_iv: "d53e1cad15bf84ef4561d879cf93a08d"
server_key_version: "encryptionkey@0"
I20260927 05:13:25.630185 6731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:25.657471 6731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:25.662953 6731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:25.663302 6731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:25.664995 6731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:25.666245 6731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:25.666313 6731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.666450 6731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:25.666510 6731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:25.698673 6731 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44501
I20260927 05:13:25.698719 6858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44501 every 8 connection(s)
I20260927 05:13:25.699941 6731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:25.709771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6731
I20260927 05:13:25.710000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:25.710443 388 external_mini_cluster.cc:1428] Setting key cc879947da3861edea5ff4973c31874b
Sep 27 05:13:25 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:25.719478 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:25.702548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33883 (local address 127.0.97.62:41801)
0927 05:13:25.702722 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:25.702726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:25.703974 (+ 1248us) server_negotiation.cc:407] Connection header received
0927 05:13:25.705762 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:25.705768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:25.705854 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:25.705972 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:25.707448 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.708165 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.709649 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.710248 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.710366 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:25.714050 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:25.714055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:25.714059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:25.714086 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:25.716644 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:25.717317 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:25.717328 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:25.717331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:25.717449 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:25.717871 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:25.717875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:25.717878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:25.718127 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:25.718277 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:25.718892 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 05:13:25.719151 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:13:25.720358 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:25.703137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:33883)
0927 05:13:25.703807 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:25.703850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:25.705507 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:25.706196 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:25.706204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:25.706668 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:25.707286 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:25.707304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.708309 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.708315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:25.709500 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:25.709510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:25.710440 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:25.710447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:25.710713 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:25.711497 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:25.711523 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:25.713844 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:25.716777 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:25.716785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:25.716800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:25.717194 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:25.717572 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:25.717577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:25.717580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:25.717752 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:25.718252 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:25.718259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:25.718568 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 05:13:25.718804 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 05:13:25.719315 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":266,"threads_started":1}
I20260927 05:13:25.721701 6859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:25.722015 6859 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:25.722798 6859 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
I20260927 05:13:25.724396 6536 ts_manager.cc:194] Registered new tserver with Master: d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501)
I20260927 05:13:25.725203 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33883
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:13:25.754009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:26.063164 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:26.063411 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:26.063499 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:26.070276 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:26.070362 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:26.070465 6864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:26.079730 6864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:26.080857 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:26.081985 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:26.094532 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.094497 6870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.094601 6869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:26.095324 6864 server_base.cc:1034] running on GCE node
I20260927 05:13:26.095893 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:26.096668 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:26.097855 6864 hybrid_clock.cc:648] HybridClock initialized: now 1790486006097818 us; error 29 us; skew 500 ppm
I20260927 05:13:26.098143 6864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:26.105038 6864 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:13:26.106415 6864 webserver.cc:460] Webserver started at http://127.0.97.3:38475/ using document root <none> and password file <none>
I20260927 05:13:26.107122 6864 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:26.107262 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:26.107551 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:26.110323 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f8a1691fbf04196a2575a07915fa7bd"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "cd9c103d3d98a9b959b3183a494e495f"
server_key_iv: "ef1409bb20de62c2a0e29c3450e1477a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.111167 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f8a1691fbf04196a2575a07915fa7bd"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "cd9c103d3d98a9b959b3183a494e495f"
server_key_iv: "ef1409bb20de62c2a0e29c3450e1477a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.115759 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:13:26.118796 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.120023 6864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 05:13:26.120201 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "6f8a1691fbf04196a2575a07915fa7bd"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "cd9c103d3d98a9b959b3183a494e495f"
server_key_iv: "ef1409bb20de62c2a0e29c3450e1477a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.120396 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:26.134624 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:26.138273 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:26.138581 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:26.139611 6864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:26.140944 6864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:26.141012 6864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.141101 6864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:26.141145 6864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.173003 6864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43191
I20260927 05:13:26.173094 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43191 every 8 connection(s)
I20260927 05:13:26.174233 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:26.179768 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6864
I20260927 05:13:26.179971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:26.180400 388 external_mini_cluster.cc:1428] Setting key e7b63a1717b283937399321063646375
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:26.193501 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.176660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55049 (local address 127.0.97.62:41801)
0927 05:13:26.176872 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.176878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.178083 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 05:13:26.180050 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.180055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.180142 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.180261 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.181857 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.182826 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.184408 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.184807 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.184950 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.188210 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.188217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.188222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.188262 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.190630 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.191384 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.191390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.191393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.191522 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.191935 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.191939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.191943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.192208 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.192354 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.192972 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.193171 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:26.194594 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:55049)
0927 05:13:26.177921 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.177959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.179697 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.180500 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.180512 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.181026 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.181700 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.181716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.182989 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.182994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.184245 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.184256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.185013 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.185022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.185267 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.185923 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.185944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.188012 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.190785 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.190793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.190807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.191264 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.191641 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.191647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.191650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.191821 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.192328 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.192335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.192630 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.192863 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.193317 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":276,"threads_started":1}
I20260927 05:13:26.195866 6992 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:26.196103 6992 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:26.196697 6992 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
I20260927 05:13:26.197855 6536 ts_manager.cc:194] Registered new tserver with Master: 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:26.198562 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55049
I20260927 05:13:26.207237 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260927 05:13:26.246701 6725 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 05:13:26.252336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46223
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41801,127.0.97.61:46223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:26.558708 6997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:26.558954 6997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:26.559105 6997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:26.566021 6997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:26.566109 6997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:26.566154 6997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:26.566208 6997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:26.566244 6997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:26.575870 6997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41801,127.0.97.61:46223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
W20260927 05:13:26.576133 6997 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:41801,127.0.97.61:46223'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 05:13:26.577104 6997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:26.578279 6997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:26.589564 7005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.589720 7002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.589694 7003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:26.590677 6997 server_base.cc:1034] running on GCE node
I20260927 05:13:26.591559 6997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:26.592852 6997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:26.594123 6997 hybrid_clock.cc:648] HybridClock initialized: now 1790486006594051 us; error 57 us; skew 500 ppm
I20260927 05:13:26.594463 6997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:26.597410 6997 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 05:13:26.598726 6997 webserver.cc:460] Webserver started at http://127.0.97.61:43747/ using document root <none> and password file <none>
I20260927 05:13:26.599434 6997 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:26.599567 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:26.599853 6997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:26.602700 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/instance:
uuid: "38b24a74000040b88f7a78ce32933c67"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "3900d6c6e8d772a1702b372ce180a733"
server_key_iv: "5e5524a9384044cbd4f193d82fcb536a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.603462 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal/instance:
uuid: "38b24a74000040b88f7a78ce32933c67"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "3900d6c6e8d772a1702b372ce180a733"
server_key_iv: "5e5524a9384044cbd4f193d82fcb536a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.608327 6997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 05:13:26.611545 7012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.612816 6997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:13:26.612984 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "38b24a74000040b88f7a78ce32933c67"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "3900d6c6e8d772a1702b372ce180a733"
server_key_iv: "5e5524a9384044cbd4f193d82fcb536a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.613184 6997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:26.625612 6997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:26.629143 6997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:26.629387 6997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:26.646612 6997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46223
I20260927 05:13:26.646627 7063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46223 every 8 connection(s)
I20260927 05:13:26.647934 6997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:13:26.651273 7064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:26.657307 7064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:13:26.658157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 6997
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:13:26.677853 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.659862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.0.97.62:41801)
0927 05:13:26.660054 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.660060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.661025 (+ 965us) server_negotiation.cc:407] Connection header received
0927 05:13:26.662845 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.662850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.662929 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.663077 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.664105 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.664837 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.666269 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.666682 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.666811 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.670620 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.670630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.670636 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.670677 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.674030 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.674939 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.674974 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.674977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.675140 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.675660 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.675663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.675665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.676104 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.676242 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.677200 (+ 958us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.677508 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:13:26.678892 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.660266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37068)
0927 05:13:26.660877 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.660914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.662616 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.663125 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.663132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.663647 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.663941 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.663956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.664966 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.664974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.666127 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.666137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.666893 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.666899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.667282 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.667921 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.667942 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.670426 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.674164 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.674172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.674179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.674698 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.675276 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.675280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.675283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.675542 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.676248 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.676256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.676730 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.677042 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.677500 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":62,"thread_start_us":263,"threads_started":1}
I20260927 05:13:26.678913 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.661226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.0.97.61:46223)
0927 05:13:26.661835 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.661841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.661873 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:13:26.662628 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.662647 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.662841 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.663062 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.664476 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.665617 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.666787 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.667480 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.667623 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.671981 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.671991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.672004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.672043 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.675290 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.675963 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.675970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.675974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.676111 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.676566 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.676571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.676574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.677033 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.677374 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.677893 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.678133 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":438,"thread_start_us":303,"threads_started":1}
I20260927 05:13:26.680641 7064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:13:26.691581 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.681254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44504)
0927 05:13:26.681402 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.681421 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.681556 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.681811 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.681814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.682083 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.682207 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.682213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.683154 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.683157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.684004 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.684013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.684639 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.684643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.684755 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.685301 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.685316 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.687088 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.689628 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.689632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.689634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.689952 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.690264 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.690267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.690270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.690396 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.690823 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.690826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.690937 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.691124 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.691308 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:13:26.691658 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.681373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.97.61:46223)
0927 05:13:26.681539 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.681545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.681562 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:26.681648 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.681652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.681713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.681805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.682309 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.683052 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.684112 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.684523 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.684616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.687236 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.687241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.687244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.687268 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.689524 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.690047 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.690051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.690053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.690146 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.690496 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.690498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.690500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.690729 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.690824 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.691200 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.691421 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":47}
I20260927 05:13:26.695874 7064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Bootstrap starting.
I20260927 05:13:26.698881 7064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:26.700104 7064 log.cc:824] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:26.702852 7064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: No bootstrap required, opened a new log
I20260927 05:13:26.709841 7064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } }
I20260927 05:13:26.710106 7064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:26.710175 7064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b24a74000040b88f7a78ce32933c67, State: Initialized, Role: FOLLOWER
I20260927 05:13:26.710609 7064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } }
I20260927 05:13:26.711323 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } }
I20260927 05:13:26.711658 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:26.712214 7064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:13:26.720773 7081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:13:26.720923 7081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:13:26.723366 6997 master_runner.cc:428] Detected that this master 38b24a74000040b88f7a78ce32933c67 is joining an existing cluster
I20260927 05:13:26.723462 6997 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:46223
I20260927 05:13:26.728431 6859 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
I20260927 05:13:26.745283 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.727065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.97.62:41801)
0927 05:13:26.727260 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.727269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.727688 (+ 419us) server_negotiation.cc:407] Connection header received
0927 05:13:26.727904 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.727909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.727987 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.728058 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.730193 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.731336 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.733512 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.734239 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.734366 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.737126 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.737134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.737142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.737183 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.741132 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.743323 (+ 2191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.743330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.743334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.743478 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.743863 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.743868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.743871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.744154 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.744306 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.744722 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.744922 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:13:26.745422 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.727044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37074)
0927 05:13:26.727583 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.727605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.727747 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.728848 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.728855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.729567 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.729986 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.730000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.731507 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.731525 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.733378 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.733391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.734680 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.734690 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.734917 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.735650 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.735682 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.736915 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.741335 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.741342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.741347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.743218 (+ 1871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.743600 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.743604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.743608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.743748 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.744273 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.744278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.744468 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.744611 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.744901 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":45,"thread_start_us":309,"threads_started":1}
I20260927 05:13:26.748787 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.727880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.0.97.61:46223)
0927 05:13:26.728017 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.728024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.728583 (+ 559us) server_negotiation.cc:407] Connection header received
0927 05:13:26.728813 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.728819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.728885 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.728988 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.730317 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.731373 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.734945 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.736087 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.736187 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.738842 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.738848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.738852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.738887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.742958 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.746684 (+ 3726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.746689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.746692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.746819 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.747224 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.747230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.747233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.747642 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.747751 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.748201 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.748397 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260927 05:13:26.749802 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.727817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44508)
0927 05:13:26.728493 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.728511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.728666 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.729146 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.729152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.729827 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.730176 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.730187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.731623 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.731627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.734809 (+ 3182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.734822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.736490 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.736495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.736656 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.737299 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.737319 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.738659 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.746011 (+ 7352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.746016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.746020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.746551 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.746944 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.746949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.746953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.747117 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.747827 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.747831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.747956 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.748995 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.749310 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":1397,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:13:26.768591 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.752814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.0.97.61:46223)
0927 05:13:26.753060 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.753068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.755695 (+ 2627us) server_negotiation.cc:407] Connection header received
0927 05:13:26.756015 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.756020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.756093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.756201 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.757173 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.757929 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.759413 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.759877 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.760013 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.763116 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.763122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.763133 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.763160 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.765736 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.766428 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.766434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.766437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.766569 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.766937 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.766943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.766946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.767353 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.767525 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.768090 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.768325 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 05:13:26.768693 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.752762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44514)
0927 05:13:26.755580 (+ 2818us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.755614 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.755811 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.756241 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.756248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.756828 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.757035 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.757044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.758123 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.758128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.759261 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.759272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.760080 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.760122 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.760275 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.760934 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.760960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.762914 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.765866 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.765879 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.765887 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.766292 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.766680 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.766688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.766691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.766824 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.767507 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.767523 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.767737 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.767945 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.768251 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2609,"thread_start_us":2466,"threads_started":1}
I20260927 05:13:26.770555 6536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:46223: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "38b24a74000040b88f7a78ce32933c67"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 46223
}
attrs {
promote: true
}
}
dest_uuid: "06b69ab300d2440d8a05e4fac552aab8"
cas_config_opid_index: -1
I20260927 05:13:26.771740 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } }
I20260927 05:13:26.784881 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 38b24a74000040b88f7a78ce32933c67 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:26.786679 7039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:26.789161 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:46223
I20260927 05:13:26.789629 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:26.789924 6997 master.cc:537] Master@127.0.97.61:46223 shutting down...
I20260927 05:13:26.790887 7039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 05:13:26.791285 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:26.791937 7093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.60.keytab.
W20260927 05:13:26.795639 6523 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 05:13:26.796361 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44051
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:13:26.810112 6997 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:26.810491 6997 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:26.810679 6997 tablet_replica.cc:330] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: stopping tablet replica
W20260927 05:13:26.813809 6523 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 -> Peer 38b24a74000040b88f7a78ce32933c67 (127.0.97.61:46223): Couldn't send request to peer 38b24a74000040b88f7a78ce32933c67. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 05:13:26.816380 6997 master.cc:552] Master@127.0.97.61:46223 shutdown complete.
I20260927 05:13:26.816470 6997 master_runner.cc:305] Clearing existing system tablet
I20260927 05:13:26.816820 6997 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:26.816921 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:26.819157 6997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.001s
I20260927 05:13:26.820542 7099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.820838 6997 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:26.821112 6997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 05:13:26.821198 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "38b24a74000040b88f7a78ce32933c67"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "3900d6c6e8d772a1702b372ce180a733"
server_key_iv: "5e5524a9384044cbd4f193d82fcb536a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.824749 6997 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:26.829089 6997 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 05:13:26.829278 6997 log.cc:1196] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 05:13:26.829648 6997 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Deleting consensus metadata
I20260927 05:13:26.829883 6997 master_runner.cc:315] Copying system tablet from 127.0.97.62:41801
I20260927 05:13:26.832247 6997 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41801
I20260927 05:13:26.844405 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.832904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.97.62:41801)
0927 05:13:26.833116 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:26.833123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:26.833564 (+ 441us) server_negotiation.cc:407] Connection header received
0927 05:13:26.833831 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:26.833836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:26.833920 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:26.834019 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:26.834830 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.835622 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.836817 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.837210 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.837355 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.839046 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:26.839053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:26.839058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:26.839094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:26.841694 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.842270 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.842276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.842279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.842400 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:26.843032 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:26.843039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:26.843043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:26.843386 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:26.843531 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:26.843865 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 05:13:26.844065 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:13:26.844482 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:26.832896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37084)
0927 05:13:26.833466 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:26.833485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:26.833627 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:26.834044 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:26.834048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:26.834462 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:26.834689 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.834698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.835753 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.835758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.836674 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:26.836685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:26.837364 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:26.837369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:26.837558 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:26.838172 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:26.838188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:26.838830 (+ 642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:26.841814 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.841821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.841825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.842158 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.842771 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:26.842776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:26.842780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:26.842912 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:26.843495 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:26.843500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:26.843650 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 05:13:26.843782 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:13:26.844056 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":336,"threads_started":1}
I20260927 05:13:26.845463 6556 tablet_copy_service.cc:140] P 06b69ab300d2440d8a05e4fac552aab8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 38b24a74000040b88f7a78ce32933c67 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37084)
I20260927 05:13:26.845716 6556 tablet_copy_service.cc:161] P 06b69ab300d2440d8a05e4fac552aab8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 38b24a74000040b88f7a78ce32933c67 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37084: session id = 38b24a74000040b88f7a78ce32933c67-00000000000000000000000000000000
I20260927 05:13:26.848731 6556 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 05:13:26.850375 6997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 05:13:26.854940 6997 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: tablet copy: Starting download of 0 data blocks...
I20260927 05:13:26.855252 6997 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: tablet copy: Starting download of 1 WAL segments...
I20260927 05:13:26.857369 6997 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 05:13:26.860332 6556 tablet_copy_service.cc:342] P 06b69ab300d2440d8a05e4fac552aab8: Request end of tablet copy session 38b24a74000040b88f7a78ce32933c67-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37084
I20260927 05:13:26.860502 6556 tablet_copy_service.cc:433] P 06b69ab300d2440d8a05e4fac552aab8: ending tablet copy session 38b24a74000040b88f7a78ce32933c67-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 38b24a74000040b88f7a78ce32933c67
I20260927 05:13:26.871464 6997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:26.876569 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.880306 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:26.887607 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:26.892417 6997 server_base.cc:1034] running on GCE node
I20260927 05:13:26.892755 6997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:26.893399 6997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:26.894642 6997 hybrid_clock.cc:648] HybridClock initialized: now 1790486006894559 us; error 48 us; skew 500 ppm
I20260927 05:13:26.894877 6997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:26 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:26.897128 6997 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 05:13:26.898391 6997 webserver.cc:460] Webserver started at http://127.0.97.61:36257/ using document root <none> and password file <none>
I20260927 05:13:26.898722 6997 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:26.898798 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:26.901819 6997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:13:26.903658 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:26.904064 6997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260927 05:13:26.904322 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "38b24a74000040b88f7a78ce32933c67"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "3900d6c6e8d772a1702b372ce180a733"
server_key_iv: "5e5524a9384044cbd4f193d82fcb536a"
server_key_version: "encryptionkey@0"
I20260927 05:13:26.904553 6997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:26.919930 6997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:26.923758 6997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:26.948104 6997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46223
I20260927 05:13:26.948174 7167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46223 every 8 connection(s)
I20260927 05:13:26.949177 6997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:13:26.955090 7168 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:13:26.958768 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Bootstrap starting.
I20260927 05:13:26.974261 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:13:26.974865 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: Bootstrap complete.
I20260927 05:13:26.976044 7168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } }
I20260927 05:13:26.976405 7168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38b24a74000040b88f7a78ce32933c67, State: Initialized, Role: LEARNER
I20260927 05:13:26.976682 7168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } }
I20260927 05:13:26.977612 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:26.977881 7168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:26.978022 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: LEARNER
I20260927 05:13:26.986553 7182 catalog_manager.cc:1259] Loaded cluster ID: da3c825fa4fe44899b55c976c03a84de
I20260927 05:13:26.986665 7182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: loading cluster ID for follower catalog manager: success
I20260927 05:13:26.990613 7182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: acquiring CA information for follower catalog manager: success
I20260927 05:13:26.992931 7182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:13:27.159031 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:27.159300 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:27.159444 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:27.166725 7097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:27.166815 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:27.166893 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:27.166989 7097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:27.167197 7097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:27.177423 7097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:27.178807 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:27.180133 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:27.191929 7187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:27.191941 7190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:27.191987 7188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:27.198169 7097 server_base.cc:1034] running on GCE node
I20260927 05:13:27.198897 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:27.200189 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:27.200861 6992 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
I20260927 05:13:27.201431 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790486007201391 us; error 43 us; skew 500 ppm
I20260927 05:13:27.201773 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:27.205284 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 05:13:27.206885 7097 webserver.cc:460] Webserver started at http://127.0.97.60:37383/ using document root <none> and password file <none>
I20260927 05:13:27.207582 7097 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:27.207707 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:27.207994 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:27.211265 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/instance:
uuid: "973da5290c9e497491e50cd4f8f34d09"
format_stamp: "Formatted at 2026-09-27 05:13:27 on dist-test-slave-1t6p"
server_key: "019329aff1ab53473b3bb91841e3c2be"
server_key_iv: "4805446d70a7d26504973e44340c3cb7"
server_key_version: "encryptionkey@0"
I20260927 05:13:27.212002 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal/instance:
uuid: "973da5290c9e497491e50cd4f8f34d09"
format_stamp: "Formatted at 2026-09-27 05:13:27 on dist-test-slave-1t6p"
server_key: "019329aff1ab53473b3bb91841e3c2be"
server_key_iv: "4805446d70a7d26504973e44340c3cb7"
server_key_version: "encryptionkey@0"
I20260927 05:13:27.216917 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:13:27.220292 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:27.221573 7097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:13:27.221730 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "973da5290c9e497491e50cd4f8f34d09"
format_stamp: "Formatted at 2026-09-27 05:13:27 on dist-test-slave-1t6p"
server_key: "019329aff1ab53473b3bb91841e3c2be"
server_key_iv: "4805446d70a7d26504973e44340c3cb7"
server_key_version: "encryptionkey@0"
I20260927 05:13:27.221896 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:27.235656 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:27.240167 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:27.240477 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:27.259232 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44051
I20260927 05:13:27.259681 7248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44051 every 8 connection(s)
I20260927 05:13:27.260572 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:13:27.264083 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:27.270409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 7097
I20260927 05:13:27.270391 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:13:27.292829 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.273668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.97.62:41801)
0927 05:13:27.273854 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.273861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.274329 (+ 468us) server_negotiation.cc:407] Connection header received
0927 05:13:27.275360 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.275365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.275435 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.275543 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.277526 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.278724 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.280734 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.281381 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.281539 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.285500 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.285506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.285510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.285538 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.289386 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.290175 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.290182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.290186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.290326 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.291138 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.291143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.291146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.291486 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.291616 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.292298 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.292510 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 05:13:27.293885 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.272567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.0.97.60:44051)
0927 05:13:27.273222 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.273231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.273265 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:13:27.275107 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.275128 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.275387 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.275589 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.277722 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.279314 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.280964 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.281810 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.281955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.286528 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.286539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.286553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.286590 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.290321 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.291138 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.291146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.291149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.291283 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.291648 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.291652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.291656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.291981 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.292212 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.292588 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.292795 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
I20260927 05:13:27.293922 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.273522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37088)
0927 05:13:27.274168 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.274218 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.275126 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.275644 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.275650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.276870 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.277363 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.277384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.278849 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.278853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.280589 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.280619 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.281639 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.281648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.281952 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.282731 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.282763 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.285326 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.289524 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.289532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.289540 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.290044 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.290786 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.290791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.290795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.291027 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.291610 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.291619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.291911 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.292172 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.292639 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":24,"spinlock_wait_cycles":13184,"thread_start_us":363,"threads_started":1}
I20260927 05:13:27.296185 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:13:27.308369 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.296783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44528)
0927 05:13:27.296922 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.296946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.297017 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.297645 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.297649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.297970 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.298080 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.298088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.299405 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.299411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.300267 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.300277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.300962 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.300968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.301152 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.301607 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.301623 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.303411 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.306133 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.306138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.306142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.306475 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.306838 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.306843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.306847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.306969 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.307477 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.307483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.307630 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.307760 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.308016 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 05:13:27.308382 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.296941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.97.61:46223)
0927 05:13:27.297390 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.297394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.297408 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:13:27.297452 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.297455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.297508 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.297629 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.298261 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.299274 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.300414 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.300814 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.300938 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.303585 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.303589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.303593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.303626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.305994 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.306588 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.306595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.306605 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.306733 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.307107 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.307112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.307115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.307352 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.307520 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.307841 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.308035 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
I20260927 05:13:27.309398 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:13:27.320814 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.310086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.0.1:36476)
0927 05:13:27.310234 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.310251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.310328 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.310562 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.310565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.310837 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.310955 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.310961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.311850 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.311853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.312629 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.312638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.313389 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.313393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.313545 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.314384 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.314400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.316251 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.318858 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.318861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.318864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.319255 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.319608 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.319611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.319613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.319746 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.320190 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.320193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.320300 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.320396 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.320562 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:13:27.321005 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.310158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.97.60:44051)
0927 05:13:27.310301 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.310305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.310320 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:13:27.310408 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.310411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.310457 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.310533 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.311120 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.311753 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.312741 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.313270 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.313360 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.316408 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.316412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.316415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.316440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.318737 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.319360 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.319364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.319366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.319489 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.319846 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.319849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.319851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.320085 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.320186 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.320494 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.320654 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":51}
I20260927 05:13:27.326074 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Bootstrap starting.
I20260927 05:13:27.329087 7249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:27.330044 7249 log.cc:824] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:27.332912 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: No bootstrap required, opened a new log
I20260927 05:13:27.338968 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.333039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44536)
0927 05:13:27.333452 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.333466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.333621 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.334000 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.334004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.334030 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:27.334227 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.334235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.335648 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.335652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.336863 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.336874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.338435 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.338439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.338562 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.338598 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.338667 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.338749 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":243,"threads_started":1}
I20260927 05:13:27.339080 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.333135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.0.97.61:46223)
0927 05:13:27.333352 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.333358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.333544 (+ 186us) server_negotiation.cc:407] Connection header received
0927 05:13:27.333741 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.333747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.333837 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.333943 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:27.334425 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.335512 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.337014 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.338288 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.338421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.338601 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.338704 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.338826 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 05:13:27.340005 7143 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 05:13:27.340570 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b69ab300d2440d8a05e4fac552aab8. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:27.340830 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: LEARNER
I20260927 05:13:27.341732 7249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } }
I20260927 05:13:27.342036 7249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:27.342170 7249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973da5290c9e497491e50cd4f8f34d09, State: Initialized, Role: FOLLOWER
I20260927 05:13:27.342672 7249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } }
I20260927 05:13:27.343446 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } } }
I20260927 05:13:27.343719 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:27.344419 7249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:13:27.354372 7269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:13:27.354511 7269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:13:27.358151 7097 master_runner.cc:428] Detected that this master 973da5290c9e497491e50cd4f8f34d09 is joining an existing cluster
I20260927 05:13:27.358250 7097 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44051
I20260927 05:13:27.373227 7258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:27.373540 7258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } }
I20260927 05:13:27.375144 7258 leader_election.cc:288] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801), 38b24a74000040b88f7a78ce32933c67 (127.0.97.61:46223)
I20260927 05:13:27.376361 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973da5290c9e497491e50cd4f8f34d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b69ab300d2440d8a05e4fac552aab8" is_pre_election: true
I20260927 05:13:27.377152 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973da5290c9e497491e50cd4f8f34d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b24a74000040b88f7a78ce32933c67" is_pre_election: true
I20260927 05:13:27.378408 7201 leader_election.cc:302] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 973da5290c9e497491e50cd4f8f34d09; no voters: 06b69ab300d2440d8a05e4fac552aab8, 38b24a74000040b88f7a78ce32933c67
I20260927 05:13:27.379154 7258 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 05:13:27.380053 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.362208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.97.60:44051)
0927 05:13:27.362377 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.362383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.363509 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 05:13:27.363706 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.363714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.363814 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.363941 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.365213 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.366720 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.368782 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.369583 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.369708 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.372827 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.372836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.372844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.372886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.376829 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.377570 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.377574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.377577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.377757 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.378212 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.378217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.378219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.378576 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.378721 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.379391 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.379622 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:13:27.380954 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.362253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.0.1:36478)
0927 05:13:27.363396 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.363431 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.363560 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.363975 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.363980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.364682 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.365056 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.365067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.366938 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.366948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.368617 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.368635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.369972 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.369994 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.370259 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.371155 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.371191 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.372611 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.376971 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.376975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.376978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.377436 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.377869 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.377872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.377875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.378050 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.378774 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.378778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.378946 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.379105 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.380412 (+ 1307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":892,"threads_started":1}
I20260927 05:13:27.391177 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.361901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.97.62:41801)
0927 05:13:27.362083 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.362089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.369333 (+ 7244us) server_negotiation.cc:407] Connection header received
0927 05:13:27.369569 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.369575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.369651 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.369717 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.371238 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.372337 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.382188 (+ 9851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.382923 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.383033 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.385673 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.385681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.385688 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.385730 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.388487 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.389201 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.389206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.389210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.389366 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.389800 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.389804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.389806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.390032 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.390188 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.390592 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.390805 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:13:27.391234 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.362291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37102)
0927 05:13:27.369242 (+ 6951us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.369260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.369430 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.369911 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.369917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.370656 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.370990 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.371026 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.379476 (+ 8450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.379481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.382029 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.382042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.383257 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.383264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.383436 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.384050 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.384066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.385447 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.388630 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.388634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.388638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.389067 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.389490 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.389498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.389504 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.389666 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.390168 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.390173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.390358 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.390491 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.390778 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6827,"thread_start_us":6655,"threads_started":1}
I20260927 05:13:27.395082 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.369484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.97.61:46223)
0927 05:13:27.369688 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.369694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.381248 (+ 11554us) server_negotiation.cc:407] Connection header received
0927 05:13:27.381705 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.381710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.381787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.381856 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.382900 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.383715 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.386098 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.386739 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.386833 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.389013 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.389019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.389024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.389058 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.392358 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.393167 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.393173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.393176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.393321 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.393692 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.393701 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.393704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.393922 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.394082 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.394497 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.394718 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:13:27.395138 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.369838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44548)
0927 05:13:27.381169 (+ 11331us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.381186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.381539 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.382068 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.382075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.382561 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.382742 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.382752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.383858 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.383863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.385900 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.385916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.387076 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.387082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.387268 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.387746 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.387767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.388817 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.392518 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.392523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.392527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.393035 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.393460 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.393463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.393465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.393567 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.394065 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.394070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.394195 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.394365 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.394688 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11238,"thread_start_us":11464,"threads_started":1}
I20260927 05:13:27.399269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6597
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 05:13:27.411919 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.396671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.0.1:36486)
0927 05:13:27.396909 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.396928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.397034 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.397371 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.397376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.397808 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.397971 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.397982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.399105 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.399111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.400465 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.400488 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.401642 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.401648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.401810 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.402425 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.402447 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.405030 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.409325 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.409330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.409334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.409808 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.410218 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.410223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.410226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.410399 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.410987 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.410992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.411189 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.411326 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.411578 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 05:13:27.411926 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.396665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.97.60:44051)
0927 05:13:27.396804 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.396808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.396991 (+ 183us) server_negotiation.cc:407] Connection header received
0927 05:13:27.397155 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.397159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.397238 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.397309 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.398135 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.398933 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.400640 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.401325 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.401446 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.405226 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.405237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.405244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.405276 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.409177 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.409947 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.409954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.409958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.410094 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.410548 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.410554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.410557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.410856 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.411006 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.411404 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.411585 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:13:27.413543 6536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "973da5290c9e497491e50cd4f8f34d09"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44051
}
attrs {
promote: true
}
}
dest_uuid: "06b69ab300d2440d8a05e4fac552aab8"
cas_config_opid_index: 5
I20260927 05:13:27.414541 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
I20260927 05:13:27.416539 7256 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.418936 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.400207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.0.97.1:33065)
0927 05:13:27.400777 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.400786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.400814 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:13:27.400878 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.400885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.401100 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.401279 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.402607 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.404005 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.405462 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.406168 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.406368 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.410763 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.410775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.410792 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.410833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.414548 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.415248 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.415255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.415257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.415372 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.416477 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.416482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.416485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.416893 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.418181 (+ 1288us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.418326 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.418563 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":536,"threads_started":1}
I20260927 05:13:27.419858 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 05:13:27.419869 7224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:27.420526 6720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:27.420403 7280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 05:13:27.421680 7256 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44051
I20260927 05:13:27.422811 7224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 05:13:27.422753 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.423519 7256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:27.424294 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:27.433697 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.439194 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.439492 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: LEARNER
I20260927 05:13:27.439757 7279 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: attempting to promote NON_VOTER 38b24a74000040b88f7a78ce32933c67 to VOTER
I20260927 05:13:27.440538 7279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
I20260927 05:13:27.442787 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 LEARNER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 05:13:27.442934 7223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 05:13:27.443303 7097 master.cc:537] Master@127.0.97.60:44051 shutting down...
I20260927 05:13:27.443336 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 05:13:27.444828 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:27.447181 7279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 38b24a74000040b88f7a78ce32933c67 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
W20260927 05:13:27.458032 6521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 -> Peer 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60:44051): Couldn't send request to peer 973da5290c9e497491e50cd4f8f34d09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 05:13:27.459794 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 38b24a74000040b88f7a78ce32933c67 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.463747 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.463986 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:27.465631 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.465900 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:27.466423 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.466655 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:27.467249 7224 raft_consensus.cc:858] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
W20260927 05:13:27.467427 7224 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
I20260927 05:13:27.482060 7224 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 38b24a74000040b88f7a78ce32933c67 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:27.486831 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:27.487146 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 05:13:27.487336 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:27.487524 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 05:13:27.487440 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b69ab300d2440d8a05e4fac552aab8. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: true } } }
I20260927 05:13:27.487645 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 05:13:27.490509 7097 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 05:13:27.490952 7097 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 05:13:27.491489 7097 tablet_replica.cc:330] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: stopping tablet replica
I20260927 05:13:27.509554 7097 master.cc:552] Master@127.0.97.60:44051 shutdown complete.
I20260927 05:13:27.509655 7097 master_runner.cc:305] Clearing existing system tablet
I20260927 05:13:27.510036 7097 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:27.510128 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:27.513049 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 05:13:27.515349 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:27.515749 7097 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:27.515995 7097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:27.516090 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "973da5290c9e497491e50cd4f8f34d09"
format_stamp: "Formatted at 2026-09-27 05:13:27 on dist-test-slave-1t6p"
server_key: "019329aff1ab53473b3bb91841e3c2be"
server_key_iv: "4805446d70a7d26504973e44340c3cb7"
server_key_version: "encryptionkey@0"
I20260927 05:13:27.520726 7097 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:27.527520 7097 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 05:13:27.527667 7097 log.cc:1196] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 05:13:27.528122 7097 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Deleting consensus metadata
I20260927 05:13:27.528349 7097 master_runner.cc:315] Copying system tablet from 127.0.97.62:41801
I20260927 05:13:27.536000 7097 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41801
I20260927 05:13:27.551541 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.536779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.97.62:41801)
0927 05:13:27.537003 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.537012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.537397 (+ 385us) server_negotiation.cc:407] Connection header received
0927 05:13:27.537563 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.537569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.537651 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.537760 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:27.538740 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.539910 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.541744 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.542394 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.542528 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.544808 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:27.544815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:27.544820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:27.544859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:27.548720 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.549487 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.549492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.549496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.549639 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:27.550018 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:27.550023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:27.550026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:27.550328 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:27.550467 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.550891 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.551181 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:13:27.551584 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.536758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37118)
0927 05:13:27.537274 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.537293 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.537435 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.537801 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.537806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.538342 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:27.538582 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.538594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.540092 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.540097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.541573 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.541587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.542582 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.542589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.542755 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.543584 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:27.543619 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:27.544610 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:27.548865 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.548875 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.548880 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.549374 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.549747 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:27.549752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:27.549756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:27.549914 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:27.550444 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:27.550450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:27.550631 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.550785 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.551120 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":274,"threads_started":1}
I20260927 05:13:27.552512 6556 tablet_copy_service.cc:140] P 06b69ab300d2440d8a05e4fac552aab8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 973da5290c9e497491e50cd4f8f34d09 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37118)
I20260927 05:13:27.552683 6556 tablet_copy_service.cc:161] P 06b69ab300d2440d8a05e4fac552aab8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 973da5290c9e497491e50cd4f8f34d09 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37118: session id = 973da5290c9e497491e50cd4f8f34d09-00000000000000000000000000000000
I20260927 05:13:27.556309 6556 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 05:13:27.558152 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 05:13:27.565115 7097 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: tablet copy: Starting download of 0 data blocks...
I20260927 05:13:27.565387 7097 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: tablet copy: Starting download of 1 WAL segments...
I20260927 05:13:27.567916 7097 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 05:13:27.573751 6556 tablet_copy_service.cc:342] P 06b69ab300d2440d8a05e4fac552aab8: Request end of tablet copy session 973da5290c9e497491e50cd4f8f34d09-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37118
I20260927 05:13:27.573889 6556 tablet_copy_service.cc:433] P 06b69ab300d2440d8a05e4fac552aab8: ending tablet copy session 973da5290c9e497491e50cd4f8f34d09-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 973da5290c9e497491e50cd4f8f34d09
I20260927 05:13:27.574875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6597
I20260927 05:13:27.592207 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:27.594688 7300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:27.595851 7301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:27.599568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33065
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41801
--webserver_port=35443
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:27.599666 7303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:27.602962 7097 server_base.cc:1034] running on GCE node
I20260927 05:13:27.603314 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:27.603912 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:27.605098 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790486007605055 us; error 34 us; skew 500 ppm
I20260927 05:13:27.605362 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:27 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:27.607748 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 05:13:27.617911 7097 webserver.cc:460] Webserver started at http://127.0.97.60:33753/ using document root <none> and password file <none>
I20260927 05:13:27.618219 7097 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:27.618315 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:27.627177 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.005s
I20260927 05:13:27.632931 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:27.633419 7097 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.006s
I20260927 05:13:27.633553 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "973da5290c9e497491e50cd4f8f34d09"
format_stamp: "Formatted at 2026-09-27 05:13:27 on dist-test-slave-1t6p"
server_key: "019329aff1ab53473b3bb91841e3c2be"
server_key_iv: "4805446d70a7d26504973e44340c3cb7"
server_key_version: "encryptionkey@0"
I20260927 05:13:27.633755 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:27.654204 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:27.658215 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:27.698370 7362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44051 every 8 connection(s)
I20260927 05:13:27.698690 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44051
I20260927 05:13:27.699734 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:13:27.706940 7363 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:13:27.710698 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Bootstrap starting.
I20260927 05:13:27.726087 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:13:27.726564 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Bootstrap complete.
I20260927 05:13:27.727742 7363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
I20260927 05:13:27.728029 7363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 973da5290c9e497491e50cd4f8f34d09, State: Initialized, Role: LEARNER
I20260927 05:13:27.728340 7363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } }
I20260927 05:13:27.729135 7366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.729486 7363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:27.729693 7366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: LEARNER
I20260927 05:13:27.750068 7377 catalog_manager.cc:1259] Loaded cluster ID: da3c825fa4fe44899b55c976c03a84de
I20260927 05:13:27.750192 7377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: loading cluster ID for follower catalog manager: success
I20260927 05:13:27.753720 7377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: acquiring CA information for follower catalog manager: success
I20260927 05:13:27.756050 7377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:13:27.945108 7379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.937279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.97.60:44051)
0927 05:13:27.940071 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:27.940076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:27.940093 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:27.940153 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:27.940157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:27.940220 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:27.940355 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:27.940740 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.941702 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.943213 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.943996 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.944153 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.944392 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:27.944657 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 05:13:27.944778 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":2544,"threads_started":1}
I20260927 05:13:27.945123 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:27.937066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.0.1:36500)
0927 05:13:27.939764 (+ 2698us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:27.939786 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:27.939927 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:27.940382 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:27.940386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:27.940411 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:27.940600 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.940610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.941837 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.941845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.943062 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:27.943076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:27.944287 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:27.944293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:27.944493 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:27.944525 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:13:27.944632 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:13:27.944766 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2532,"thread_start_us":2432,"threads_started":1}
I20260927 05:13:27.946115 7338 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260927 05:13:27.946534 7366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b69ab300d2440d8a05e4fac552aab8. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:27.946846 7366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: LEARNER
W20260927 05:13:27.976114 7306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:27.976361 7306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:27.976450 7306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:27.983946 7306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:27.984038 7306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:27.984150 7306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:27.993709 7306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35443
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:27.994827 7306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:27.996074 7306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:28.008816 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:28.008862 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:28.008905 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:28.009984 7306 server_base.cc:1034] running on GCE node
I20260927 05:13:28.010533 7306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:28.011445 7306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:28.012658 7306 hybrid_clock.cc:648] HybridClock initialized: now 1790486008012613 us; error 56 us; skew 500 ppm
I20260927 05:13:28.012949 7306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:28.016294 7306 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:13:28.017567 7306 webserver.cc:460] Webserver started at http://127.0.97.1:35443/ using document root <none> and password file <none>
I20260927 05:13:28.018208 7306 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:28.018391 7306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:28.023396 7306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:28.025974 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.027274 7306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:28.027431 7306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "daa16bb930014639b04b857b99c7e5e0"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "a4b8cb42beec60edcbe7401368218a95"
server_key_iv: "2df4401c14cbf53b07018baa3eefe6bf"
server_key_version: "encryptionkey@0"
I20260927 05:13:28.028343 7306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:28.051134 7306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:28.055542 7306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:28.055800 7306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:28.056658 7306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:28.057909 7306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:28.058024 7306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.058159 7306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:28.058243 7306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.093590 7306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33065
I20260927 05:13:28.093672 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33065 every 8 connection(s)
I20260927 05:13:28.095438 7306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:28.104233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 7306
I20260927 05:13:28.104592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6731
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:13:28.139925 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.099697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52591 (local address 127.0.97.62:41801)
0927 05:13:28.112065 (+ 12368us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.112072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.112620 (+ 548us) server_negotiation.cc:407] Connection header received
0927 05:13:28.112984 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.112990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.113072 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.113160 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.116523 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.118051 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.121089 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.121901 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.122012 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.127531 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.127539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.127544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.127583 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.131260 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.136867 (+ 5607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.136875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.136881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.137032 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.137625 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.137630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.137633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.137983 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.138131 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.139351 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.139552 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12250,"thread_start_us":12374,"threads_started":1}
I20260927 05:13:28.140903 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.099764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:52591)
0927 05:13:28.112529 (+ 12765us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.112547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.112716 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.114093 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.114106 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.115271 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.116282 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.116310 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.118316 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.118324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.120907 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.120925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.122400 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.122409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.122746 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.123642 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.123684 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.127309 (+ 3625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.136103 (+ 8794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.136113 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.136129 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.136758 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.137210 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.137216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.137219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.137461 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.138504 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.138512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.138899 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.139254 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.139720 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12648,"thread_start_us":13125,"threads_started":1}
I20260927 05:13:28.142923 7508 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:28.143288 7508 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.144425 7508 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
I20260927 05:13:28.145644 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.106188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.97.2:44501)
0927 05:13:28.119772 (+ 13584us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.119784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.119820 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:13:28.119908 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.119920 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.120193 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.120369 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.121830 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.124010 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.126550 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.127501 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.128404 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.137717 (+ 9313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.137730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.137746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.137782 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.142289 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.143566 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.143574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.143577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.143734 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.144081 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.144086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.144089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.144401 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.144674 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.144983 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.145240 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13362,"thread_start_us":14321,"threads_started":1}
I20260927 05:13:28.146306 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.119554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.97.1:45427)
0927 05:13:28.125946 (+ 6392us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.125964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.126065 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.127720 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.127726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.128282 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.128656 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.128673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.130915 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.130924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.133079 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.133091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.134360 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.134367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.134542 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.135175 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.135191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.137436 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.141557 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.141561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.141567 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.142072 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.142492 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.142496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.142499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.143103 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.143791 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.143798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.145541 (+ 1743us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.145686 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.145956 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6268,"thread_start_us":6586,"threads_started":1}
I20260927 05:13:28.146474 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.114047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45427 (local address 127.0.97.61:46223)
0927 05:13:28.127218 (+ 13171us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.127224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.127248 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:13:28.127304 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.127308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.127394 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.127491 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.129071 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.130708 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.133228 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.134188 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.134329 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.137684 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.137690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.137695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.137732 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.141416 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.142231 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.142236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.142239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.142370 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.143239 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.143244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.143247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.143580 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.143745 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.145833 (+ 2088us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.146048 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13059,"thread_start_us":12952,"threads_started":1}
I20260927 05:13:28.146481 6536 ts_manager.cc:194] Re-registered known tserver with Master: daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:28.147478 6854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:28.147629 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52591
I20260927 05:13:28.148885 7509 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46223
I20260927 05:13:28.149034 7509 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.149291 7379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.099224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35405 (local address 127.0.97.60:44051)
0927 05:13:28.099416 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.099425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.109595 (+ 10170us) server_negotiation.cc:407] Connection header received
0927 05:13:28.128865 (+ 19270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.128871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.128978 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.129053 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.130214 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.131371 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.134337 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.135079 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.135179 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.141723 (+ 6544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.141731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.141737 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.141775 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.145213 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.147107 (+ 1894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.147112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.147114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.147234 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.147674 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.147680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.147688 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.148034 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.148156 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.148568 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.148845 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:13:28.149537 7509 heartbeater.cc:511] Master 127.0.97.61:46223 requested a full tablet report, sending...
I20260927 05:13:28.150552 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.099316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.97.1:35405)
0927 05:13:28.109417 (+ 10101us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.109474 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.112675 (+ 3201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.129243 (+ 16568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.129248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.129742 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.130029 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.130043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.131558 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.131563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.134121 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.134138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.135502 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.135508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.135742 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.136955 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.136981 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.140428 (+ 3447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.146519 (+ 6091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.146523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.146526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.146920 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.147361 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.147366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.147370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.147516 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.148144 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.148148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.148303 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.149699 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.150064 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9697,"thread_start_us":12822,"threads_started":1}
I20260927 05:13:28.151409 7133 ts_manager.cc:194] Registered new tserver with Master: daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:28.151669 7507 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44051
I20260927 05:13:28.151789 7507 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.152391 7507 heartbeater.cc:511] Master 127.0.97.60:44051 requested a full tablet report, sending...
I20260927 05:13:28.153746 7328 ts_manager.cc:194] Registered new tserver with Master: daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:28.232357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6731
I20260927 05:13:28.249054 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44501
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41801
--webserver_port=39083
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:13:28.504362 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: true } } }
I20260927 05:13:28.504765 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:28.506419 7521 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: attempting to promote NON_VOTER 973da5290c9e497491e50cd4f8f34d09 to VOTER
I20260927 05:13:28.506963 7521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } }
I20260927 05:13:28.508946 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 05:13:28.509017 7338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 LEARNER]: Refusing update from remote peer 06b69ab300d2440d8a05e4fac552aab8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 05:13:28.509590 7521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 05:13:28.509933 7522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 05:13:28.512882 7520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.513481 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.515830 7338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.517068 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.517300 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:28.517561 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.517791 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:28.518038 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.518276 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:28.518854 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.519228 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:28.520411 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06b69ab300d2440d8a05e4fac552aab8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:28.520717 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 05:13:28.568828 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:28.569072 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:28.569183 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:28.576053 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:28.576149 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:28.576252 7519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:28.585664 7519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39083
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:28.586798 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:28.588063 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:28.600559 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:28.600612 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:28.600610 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:28.601450 7519 server_base.cc:1034] running on GCE node
I20260927 05:13:28.601925 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:28.602700 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:28.603884 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790486008603841 us; error 40 us; skew 500 ppm
I20260927 05:13:28.604146 7519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:28.607802 7519 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:13:28.609258 7519 webserver.cc:460] Webserver started at http://127.0.97.2:39083/ using document root <none> and password file <none>
I20260927 05:13:28.609890 7519 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:28.610040 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:28.615173 7519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:28.617566 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.618868 7519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:13:28.619078 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "d42228ac01354c308bd1ff678d80d383"
format_stamp: "Formatted at 2026-09-27 05:13:25 on dist-test-slave-1t6p"
server_key: "e6adb36df0124bc7c075debd161bad61"
server_key_iv: "d53e1cad15bf84ef4561d879cf93a08d"
server_key_version: "encryptionkey@0"
I20260927 05:13:28.619990 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:28.638185 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:28.641969 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:28.642235 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:28.643147 7519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:28.644362 7519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:28.644483 7519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.644618 7519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:28.644706 7519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:28.677162 7519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44501
I20260927 05:13:28.677210 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44501 every 8 connection(s)
I20260927 05:13:28.678755 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:28.679541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 7519
I20260927 05:13:28.679800 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6864
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:28.712383 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.681155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.97.3:43191)
0927 05:13:28.685335 (+ 4180us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.685346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.688058 (+ 2712us) server_negotiation.cc:407] Connection header received
0927 05:13:28.688171 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.688183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.688470 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.688632 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.691455 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.693407 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.695935 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.697074 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.697238 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.703919 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.703934 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.703952 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.703997 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.708100 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.708917 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.708926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.708930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.709077 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.709842 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.709847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.709851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.710247 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.710533 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.711742 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.711992 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3986,"thread_start_us":3855,"threads_started":1}
I20260927 05:13:28.713658 6987 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:13:28 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486008, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:13:28.720484 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.682046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45531 (local address 127.0.97.62:41801)
0927 05:13:28.683016 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.683023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.687680 (+ 4657us) server_negotiation.cc:407] Connection header received
0927 05:13:28.690449 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.690458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.690571 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.690667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.693509 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.694962 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.707722 (+ 12760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.708431 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.708533 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.713251 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.713264 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.713273 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.713341 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.716789 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.717767 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.717771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.717775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.717922 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.718409 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.718417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.718423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.718742 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.718890 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.719858 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.720090 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":655,"threads_started":1}
I20260927 05:13:28.722779 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.682388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:45531)
0927 05:13:28.687593 (+ 5205us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.687609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.690093 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.691283 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.691297 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.692370 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.693282 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.693310 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.695250 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.695258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.707548 (+ 12290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.707560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.708816 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.708825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.709006 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.709613 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.709635 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.712772 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.716940 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.716954 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.716969 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.717651 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.718062 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.718066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.718070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.718273 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.718982 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.718991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.719405 (+ 414us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.720697 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.721203 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5063,"mutex_wait_us":33,"thread_start_us":4693,"threads_started":1}
I20260927 05:13:28.724315 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.687885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.97.2:35639)
0927 05:13:28.700472 (+ 12587us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.700499 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.700602 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.701295 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.701302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.701857 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.702598 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.702620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.704006 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.704011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.705795 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.705808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.706978 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.706986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.707340 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.709344 (+ 2004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.709371 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.715668 (+ 6297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.720338 (+ 4670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.720343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.720347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.721005 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.721403 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.721408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.721412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.721575 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.722262 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.722268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.722412 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.722555 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.722835 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12475,"mutex_wait_us":3,"thread_start_us":12347,"threads_started":1}
I20260927 05:13:28.724824 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:28.724910 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.687910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35639 (local address 127.0.97.61:46223)
0927 05:13:28.700994 (+ 13084us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.701000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.701017 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:28.701068 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.701072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.701138 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.701238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.702779 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.703848 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.705962 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.706602 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.706713 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.716680 (+ 9967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.716689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.716697 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.716760 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.720193 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.721148 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.721154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.721157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.721283 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.721704 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.721709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.721713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.722061 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.722201 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.724015 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.724559 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12971,"thread_start_us":13399,"threads_started":1}
I20260927 05:13:28.725159 7657 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.725908 7657 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
I20260927 05:13:28.726832 7658 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46223
I20260927 05:13:28.727052 7658 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.727510 7658 heartbeater.cc:511] Master 127.0.97.61:46223 requested a full tablet report, sending...
I20260927 05:13:28.727562 6536 ts_manager.cc:194] Re-registered known tserver with Master: d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501)
I20260927 05:13:28.728690 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45531
I20260927 05:13:28.728878 7133 ts_manager.cc:194] Registered new tserver with Master: d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501)
I20260927 05:13:28.728996 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.681775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56777 (local address 127.0.97.60:44051)
0927 05:13:28.684857 (+ 3082us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:28.684865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:28.687072 (+ 2207us) server_negotiation.cc:407] Connection header received
0927 05:13:28.704009 (+ 16937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:28.704014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:28.704095 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:28.704193 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:28.705628 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.707020 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.709708 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.710355 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.710454 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.717765 (+ 7311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:28.717772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:28.717776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:28.717812 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:28.721555 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.722268 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.722273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.722279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.722406 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:28.723435 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:28.723443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:28.723448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:28.723772 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:28.723938 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:28.728376 (+ 4438us) server_negotiation.cc:299] Negotiation successful
0927 05:13:28.728600 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2942,"thread_start_us":2845,"threads_started":1}
I20260927 05:13:28.731498 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:28.682387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.97.2:56777)
0927 05:13:28.686040 (+ 3653us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:28.686096 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:28.703853 (+ 17757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:28.704489 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:28.704494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:28.705026 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:28.705398 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.705411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.707205 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.707213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.709098 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:28.709111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:28.710779 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:28.710784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:28.710975 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:28.714162 (+ 3187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:28.714193 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:28.717536 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:28.721702 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.721707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.721711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.722137 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.722545 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:28.722552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:28.722557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:28.723260 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:28.724042 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:28.724047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:28.724210 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 05:13:28.729179 (+ 4969us) client_negotiation.cc:240] Negotiation successful
0927 05:13:28.729513 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3273,"mutex_wait_us":440,"thread_start_us":3005,"threads_started":1}
I20260927 05:13:28.732403 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44051
I20260927 05:13:28.732530 7656 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:28.732867 7656 heartbeater.cc:511] Master 127.0.97.60:44051 requested a full tablet report, sending...
I20260927 05:13:28.733831 7328 ts_manager.cc:194] Registered new tserver with Master: d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501)
I20260927 05:13:28.800292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6864
I20260927 05:13:28.815843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43191
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41801
--webserver_port=38475
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:29.145092 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:29.145370 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:29.145524 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:29.153075 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260927 05:13:29.153040 7508 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
W20260927 05:13:29.153172 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:29.153337 7669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:29.163110 7669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38475
--tserver_master_addrs=127.0.97.62:41801,127.0.97.61:46223,127.0.97.60:44051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:29.164287 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:29.165488 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:29.178572 7676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:29.178524 7678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:29.178658 7675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:29.179566 7669 server_base.cc:1034] running on GCE node
I20260927 05:13:29.180128 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:29.180893 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:29.182135 7669 hybrid_clock.cc:648] HybridClock initialized: now 1790486009182084 us; error 42 us; skew 500 ppm
I20260927 05:13:29.182437 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:29 dist-test-slave-1t6p krb5kdc[6491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486009, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:29.185683 7669 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:13:29.187111 7669 webserver.cc:460] Webserver started at http://127.0.97.3:38475/ using document root <none> and password file <none>
I20260927 05:13:29.187731 7669 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:29.187868 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:29.193041 7669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:13:29.195649 7685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:29.196913 7669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:29.197082 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "6f8a1691fbf04196a2575a07915fa7bd"
format_stamp: "Formatted at 2026-09-27 05:13:26 on dist-test-slave-1t6p"
server_key: "cd9c103d3d98a9b959b3183a494e495f"
server_key_iv: "ef1409bb20de62c2a0e29c3450e1477a"
server_key_version: "encryptionkey@0"
I20260927 05:13:29.198006 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:29.228406 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:29.232251 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:29.232522 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:29.233361 7669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:29.234771 7669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:29.234895 7669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:29.235033 7669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:29.235134 7669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 05:13:29.267472 7669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43191
I20260927 05:13:29.267555 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43191 every 8 connection(s)
I20260927 05:13:29.268821 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:29.274518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 7669
Sep 27 05:13:29 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486009, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 05:13:29 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486009, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:13:29 dist-test-slave-1t6p krb5kdc[6491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486009, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:13:29.314303 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.272464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.97.3:37641)
0927 05:13:29.273957 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.274011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.277198 (+ 3187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.282519 (+ 5321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.282534 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.283540 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:29.284498 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.284526 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.288678 (+ 4152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.288684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.291460 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.291471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.292744 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.292752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.293022 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.294065 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:29.294097 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:29.298181 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:29.309234 (+ 11053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.309243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.309262 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.309863 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.310322 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.310329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.310333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.310602 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.311638 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:29.311647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:29.312069 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.312351 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.312881 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":1223,"thread_start_us":821,"threads_started":1}
I20260927 05:13:29.316287 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.271636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37641 (local address 127.0.97.60:44051)
0927 05:13:29.272898 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.272905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.274167 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 05:13:29.277534 (+ 3367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.277541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.277634 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.277747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.286857 (+ 9110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.287976 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.291623 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.292220 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.292319 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.299404 (+ 7085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.299414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.299421 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.299473 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.307026 (+ 7553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.310017 (+ 2991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.310023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.310026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.310186 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.310753 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.310759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.310764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.311120 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.311254 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.315686 (+ 4432us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.315888 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1125,"thread_start_us":1014,"threads_started":1}
I20260927 05:13:29.317464 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.271673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44447 (local address 127.0.97.62:41801)
0927 05:13:29.273281 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.273289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.278081 (+ 4792us) server_negotiation.cc:407] Connection header received
0927 05:13:29.278756 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.278764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.278859 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.279023 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.286121 (+ 7098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.287638 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.290608 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.291444 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.291599 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.302314 (+ 10715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.302324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.302332 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.302398 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.309072 (+ 6674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.311324 (+ 2252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.311329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.311332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.311491 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.314663 (+ 3172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.314668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.314671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.315378 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.315514 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.316897 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.317099 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":1345,"threads_started":1}
I20260927 05:13:29.316537 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.272465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:44447)
0927 05:13:29.277999 (+ 5534us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.278022 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.278230 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.284767 (+ 6537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.284774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.285563 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:29.285951 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.285965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.288137 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.288142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.290480 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.290494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.293496 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.293502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.293704 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.294466 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:29.294515 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:29.300864 (+ 6349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:29.310524 (+ 9660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.310530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.310534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.311049 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.311599 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.311604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.311608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.311741 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.315495 (+ 3754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:29.315500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:29.315648 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.315803 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.316126 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5380,"mutex_wait_us":284,"thread_start_us":4984,"threads_started":1}
I20260927 05:13:29.319661 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44051
I20260927 05:13:29.320007 7798 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:29.320016 7799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41801
I20260927 05:13:29.320205 7799 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:29.321013 7798 heartbeater.cc:511] Master 127.0.97.60:44051 requested a full tablet report, sending...
I20260927 05:13:29.321044 7799 heartbeater.cc:511] Master 127.0.97.62:41801 requested a full tablet report, sending...
I20260927 05:13:29.321826 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.278262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.97.3:45915)
0927 05:13:29.287183 (+ 8921us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.287200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.287297 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.290059 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.290064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.290616 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:29.290939 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.290949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.293014 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.293020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.295258 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.295272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.301457 (+ 6185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.301467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.301749 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.302336 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:29.302365 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:29.305634 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:29.317379 (+ 11745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.317384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.317389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.317841 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.320043 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:29.320048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:29.320052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:29.320242 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:29.320866 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:29.320873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:29.321003 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.321168 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.321449 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8800,"thread_start_us":9097,"threads_started":1}
I20260927 05:13:29.322592 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.278260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.0.97.60:44051)
0927 05:13:29.292713 (+ 14453us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.292718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.292733 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:13:29.292774 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.292778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.292850 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.300037 (+ 7187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.300314 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.308289 (+ 7975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.311218 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.311831 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.312463 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.315192 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.315198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.315202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.315233 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.318493 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.319372 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.319376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.319379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.319532 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.320167 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.320172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.320175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.320536 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.321685 (+ 1149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.321829 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.322237 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14330,"thread_start_us":14234,"threads_started":1}
I20260927 05:13:29.322619 6536 ts_manager.cc:194] Re-registered known tserver with Master: 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:29.323035 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.278530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45915 (local address 127.0.97.61:46223)
0927 05:13:29.289755 (+ 11225us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.289761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.289779 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:13:29.289831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.289836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.289903 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.291090 (+ 1187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.291229 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.292237 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.295440 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.301227 (+ 5787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.302158 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.306013 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.306022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.306030 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.306072 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.317228 (+ 11156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.319344 (+ 2116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.319353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.319356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.319511 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.320395 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.320399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.320402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.320720 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.322027 (+ 1307us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.322373 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.322613 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11112,"thread_start_us":10997,"threads_started":1}
I20260927 05:13:29.323611 7328 ts_manager.cc:194] Registered new tserver with Master: 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:29.324847 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44447
I20260927 05:13:29.325591 7800 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46223
I20260927 05:13:29.325718 7800 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:29.327308 7800 heartbeater.cc:511] Master 127.0.97.61:46223 requested a full tablet report, sending...
I20260927 05:13:29.329401 7133 ts_manager.cc:194] Registered new tserver with Master: 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:29.331444 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.278708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.97.62:41801)
0927 05:13:29.282319 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.282324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.308084 (+ 25760us) server_negotiation.cc:407] Connection header received
0927 05:13:29.308312 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.308319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.308400 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.308485 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.313570 (+ 5085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.314796 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.318088 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.318778 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.318899 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.325680 (+ 6781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.325688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.325692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.325731 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.328542 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.329288 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.329294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.329299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.329429 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.329887 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.329896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.329900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.330300 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.330506 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.330843 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.331062 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":3416,"threads_started":1}
I20260927 05:13:29.332829 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.289750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.0.97.61:46223)
0927 05:13:29.299441 (+ 9691us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.299449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.308892 (+ 9443us) server_negotiation.cc:407] Connection header received
0927 05:13:29.309045 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.309050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.309126 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.309224 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.311683 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.312895 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.315627 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.316227 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.316346 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.326990 (+ 10644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.327028 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.327036 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.327072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.330285 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.331060 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.331065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.331068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.331222 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.331542 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.331547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.331550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.331832 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.332035 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.332342 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.332553 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9536,"thread_start_us":9712,"threads_started":1}
I20260927 05:13:29.335464 6536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:29.342293 6536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:29.366256 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.358105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.97.2:44501)
0927 05:13:29.358825 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.358836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.358865 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:13:29.359488 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.359499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.359728 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.359854 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:29.360712 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.361919 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.363611 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.365002 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.365097 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.365287 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.365844 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.366036 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":1316,"threads_started":1}
I20260927 05:13:29.366875 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.357829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44501 (local address 127.0.0.1:58852)
0927 05:13:29.358365 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.358391 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.359256 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.360135 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.360139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.360166 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:29.360503 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.360514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.362060 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.362063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.363392 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.363403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.365464 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.365470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.365650 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.365680 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.366471 (+ 791us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.366607 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"spinlock_wait_cycles":384,"thread_start_us":265,"threads_started":1}
I20260927 05:13:29.368973 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.358712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.0.1:35576)
0927 05:13:29.359306 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.359324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.359420 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.360878 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.360883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.360907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:29.361221 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.361232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.363777 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.363783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.365912 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.365940 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.368230 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.368239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.368416 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.368448 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.368541 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.368655 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":345,"threads_started":1}
I20260927 05:13:29.369395 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.359266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.0.97.3:43191)
0927 05:13:29.359822 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.359830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.359861 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:13:29.359965 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.359972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.360238 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.360444 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:29.361445 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.363556 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.366099 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.368022 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.368150 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.368343 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.368732 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.368953 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":341,"threads_started":1}
I20260927 05:13:29.370520 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 77e88228f60843a489268ddeb65a2ecc (DEFAULT_TABLE table=test-table [id=56c8eb8a2d4742cba16f90265daadfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:29.371941 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e88228f60843a489268ddeb65a2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:29.377574 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.358927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.0.97.1:33065)
0927 05:13:29.370755 (+ 11828us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.370761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.370781 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:13:29.370850 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.370854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.371099 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.371278 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:29.371843 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.373188 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.375555 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.376566 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.376708 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.376899 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.377112 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.377258 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11703,"thread_start_us":12123,"threads_started":1}
I20260927 05:13:29.377574 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.358444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:49984)
0927 05:13:29.359028 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.359046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.359265 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.371272 (+ 12007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.371278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.371302 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:29.371656 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.371667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.373400 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.373405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.375380 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.375395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.376747 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.376753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.376931 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.376975 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.377075 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.377193 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"spinlock_wait_cycles":172928,"thread_start_us":1296,"threads_started":1}
I20260927 05:13:29.378461 7733 tablet_service.cc:1467] Processing CreateTablet for tablet 77e88228f60843a489268ddeb65a2ecc (DEFAULT_TABLE table=test-table [id=56c8eb8a2d4742cba16f90265daadfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:29.379868 7733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e88228f60843a489268ddeb65a2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:29.382169 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 77e88228f60843a489268ddeb65a2ecc (DEFAULT_TABLE table=test-table [id=56c8eb8a2d4742cba16f90265daadfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:29.383281 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e88228f60843a489268ddeb65a2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:29.390743 7832 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Bootstrap starting.
I20260927 05:13:29.394076 7832 tablet_bootstrap.cc:654] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:29.394315 7833 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Bootstrap starting.
I20260927 05:13:29.395846 7832 log.cc:824] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:29.396983 7834 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Bootstrap starting.
I20260927 05:13:29.398074 7833 tablet_bootstrap.cc:654] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:29.399271 7833 log.cc:824] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:29.400264 7834 tablet_bootstrap.cc:654] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:29.401329 7834 log.cc:824] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:29.402242 7833 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: No bootstrap required, opened a new log
I20260927 05:13:29.402242 7832 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: No bootstrap required, opened a new log
I20260927 05:13:29.402499 7833 ts_tablet_manager.cc:1397] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 05:13:29.402498 7832 ts_tablet_manager.cc:1397] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260927 05:13:29.403754 7834 tablet_bootstrap.cc:492] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: No bootstrap required, opened a new log
I20260927 05:13:29.403993 7834 ts_tablet_manager.cc:1397] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 05:13:29.411399 7834 raft_consensus.cc:348] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.411733 7834 raft_consensus.cc:374] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:29.411850 7834 raft_consensus.cc:729] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa16bb930014639b04b857b99c7e5e0, State: Initialized, Role: FOLLOWER
I20260927 05:13:29.412420 7834 consensus_queue.cc:260] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.413388 7833 raft_consensus.cc:348] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.413754 7833 raft_consensus.cc:374] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:29.413833 7833 raft_consensus.cc:729] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8a1691fbf04196a2575a07915fa7bd, State: Initialized, Role: FOLLOWER
I20260927 05:13:29.413981 7834 ts_tablet_manager.cc:1428] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260927 05:13:29.414327 7833 consensus_queue.cc:260] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.414573 7832 raft_consensus.cc:348] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.415030 7832 raft_consensus.cc:374] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:29.415145 7832 raft_consensus.cc:729] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42228ac01354c308bd1ff678d80d383, State: Initialized, Role: FOLLOWER
I20260927 05:13:29.415817 7832 consensus_queue.cc:260] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.418655 7833 ts_tablet_manager.cc:1428] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Time spent starting tablet: real 0.016s user 0.005s sys 0.012s
I20260927 05:13:29.418946 7799 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
I20260927 05:13:29.423898 7832 ts_tablet_manager.cc:1428] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260927 05:13:29.424188 7657 heartbeater.cc:503] Master 127.0.97.62:41801 was elected leader, sending a full tablet report...
W20260927 05:13:29.433785 7661 tablet.cc:2369] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:29.451784 7838 raft_consensus.cc:482] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:29.452001 7838 raft_consensus.cc:504] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.453435 7838 leader_election.cc:288] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501), 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:29.459960 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.454109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44347 (local address 127.0.97.2:44501)
0927 05:13:29.454358 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.454367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.454877 (+ 510us) server_negotiation.cc:407] Connection header received
0927 05:13:29.455121 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.455127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.455192 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.455285 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:29.455767 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.456484 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.458225 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.458994 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.459199 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.459295 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.459557 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.459679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 05:13:29.460125 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.453685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44501 (local address 127.0.97.1:44347)
0927 05:13:29.454771 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.454787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.454965 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.455305 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.455310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.455335 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:29.455635 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.455642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.456629 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.456634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.458070 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.458082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.459248 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.459254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.459400 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.459423 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.459531 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.459649 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":690,"threads_started":1}
I20260927 05:13:29.460549 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.453903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46357 (local address 127.0.97.3:43191)
0927 05:13:29.454088 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.454093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.455198 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 05:13:29.455389 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.455394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.455445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.455543 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:29.456076 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.456761 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.458391 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.459199 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.459338 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.459835 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.460263 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.460391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:13:29.461047 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.453782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.97.1:46357)
0927 05:13:29.455086 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:29.455104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:29.455243 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:29.455559 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:29.455562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:29.455606 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:29.455927 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.455935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.456905 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.456912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.458241 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:29.458251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.459905 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.459911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:29.460058 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.460103 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 05:13:29.460736 (+ 633us) client_negotiation.cc:240] Negotiation successful
0927 05:13:29.460844 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":936,"threads_started":1}
I20260927 05:13:29.461006 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e88228f60843a489268ddeb65a2ecc" candidate_uuid: "daa16bb930014639b04b857b99c7e5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42228ac01354c308bd1ff678d80d383" is_pre_election: true
I20260927 05:13:29.461550 7611 raft_consensus.cc:2454] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa16bb930014639b04b857b99c7e5e0 in term 0.
I20260927 05:13:29.462041 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e88228f60843a489268ddeb65a2ecc" candidate_uuid: "daa16bb930014639b04b857b99c7e5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a1691fbf04196a2575a07915fa7bd" is_pre_election: true
I20260927 05:13:29.462208 7396 leader_election.cc:302] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d42228ac01354c308bd1ff678d80d383, daa16bb930014639b04b857b99c7e5e0; no voters:
I20260927 05:13:29.462502 7753 raft_consensus.cc:2454] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa16bb930014639b04b857b99c7e5e0 in term 0.
I20260927 05:13:29.462574 7838 raft_consensus.cc:2781] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:29.462733 7838 raft_consensus.cc:482] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:29.462857 7838 raft_consensus.cc:3037] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:29.464946 7838 raft_consensus.cc:504] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.465458 7838 leader_election.cc:288] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [CANDIDATE]: Term 1 election: Requested vote from peers d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501), 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191)
I20260927 05:13:29.465859 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e88228f60843a489268ddeb65a2ecc" candidate_uuid: "daa16bb930014639b04b857b99c7e5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42228ac01354c308bd1ff678d80d383"
I20260927 05:13:29.465876 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e88228f60843a489268ddeb65a2ecc" candidate_uuid: "daa16bb930014639b04b857b99c7e5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a1691fbf04196a2575a07915fa7bd"
I20260927 05:13:29.466029 7611 raft_consensus.cc:3037] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:29.466075 7753 raft_consensus.cc:3037] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:29.468137 7753 raft_consensus.cc:2454] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa16bb930014639b04b857b99c7e5e0 in term 1.
I20260927 05:13:29.468140 7611 raft_consensus.cc:2454] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa16bb930014639b04b857b99c7e5e0 in term 1.
I20260927 05:13:29.468503 7396 leader_election.cc:302] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d42228ac01354c308bd1ff678d80d383, daa16bb930014639b04b857b99c7e5e0; no voters:
I20260927 05:13:29.468765 7838 raft_consensus.cc:2781] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:29.469153 7838 raft_consensus.cc:686] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 1 LEADER]: Becoming Leader. State: Replica: daa16bb930014639b04b857b99c7e5e0, State: Running, Role: LEADER
I20260927 05:13:29.469650 7838 consensus_queue.cc:237] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260927 05:13:29.474325 6536 catalog_manager.cc:5495] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to daa16bb930014639b04b857b99c7e5e0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "daa16bb930014639b04b857b99c7e5e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:29.521065 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.516352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.97.1:33065)
0927 05:13:29.516531 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.516536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.516625 (+ 89us) server_negotiation.cc:407] Connection header received
0927 05:13:29.516796 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.516799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.516876 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.517041 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:29.517326 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.518036 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.519312 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.519706 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.519798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.520168 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:29.520621 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:29.520730 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.520789 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.520846 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 05:13:29.521641 7801 tablet.cc:2369] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:29.529011 7753 raft_consensus.cc:1261] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Refusing update from remote peer daa16bb930014639b04b857b99c7e5e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:29.529011 7611 raft_consensus.cc:1261] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Refusing update from remote peer daa16bb930014639b04b857b99c7e5e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:29.529961 7843 consensus_queue.cc:1035] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:29.530735 7838 consensus_queue.cc:1035] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:29.573702 6536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:29.573905 6536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:29.580399 6536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06b69ab300d2440d8a05e4fac552aab8: Sending DeleteTablet for 3 replicas of tablet 77e88228f60843a489268ddeb65a2ecc
I20260927 05:13:29.581871 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e88228f60843a489268ddeb65a2ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49984
I20260927 05:13:29.581933 7733 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e88228f60843a489268ddeb65a2ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35576
I20260927 05:13:29.582716 7856 tablet_replica.cc:330] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: stopping tablet replica
I20260927 05:13:29.582864 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e88228f60843a489268ddeb65a2ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 05:13:29.583146 7856 raft_consensus.cc:2229] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:29.583827 7856 raft_consensus.cc:2258] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:29.587839 7856 ts_tablet_manager.cc:1905] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:29.593757 7858 tablet_replica.cc:330] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: stopping tablet replica
I20260927 05:13:29.594174 7858 raft_consensus.cc:2229] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:29.594641 7858 raft_consensus.cc:2258] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:29.596683 7858 ts_tablet_manager.cc:1905] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:29.597077 7856 ts_tablet_manager.cc:1918] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:29.597242 7856 log.cc:1196] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/77e88228f60843a489268ddeb65a2ecc
I20260927 05:13:29.597784 7856 ts_tablet_manager.cc:1939] T 77e88228f60843a489268ddeb65a2ecc P daa16bb930014639b04b857b99c7e5e0: Deleting consensus metadata
I20260927 05:13:29.599383 6522 catalog_manager.cc:4841] TS daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065): tablet 77e88228f60843a489268ddeb65a2ecc (table test-table [id=56c8eb8a2d4742cba16f90265daadfda]) successfully deleted
I20260927 05:13:29.601804 7857 tablet_replica.cc:330] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: stopping tablet replica
I20260927 05:13:29.602155 7857 raft_consensus.cc:2229] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:29.602618 7857 raft_consensus.cc:2258] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:29.604389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6504
I20260927 05:13:29.604971 7857 ts_tablet_manager.cc:1905] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:29.611716 7857 ts_tablet_manager.cc:1918] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:29.611888 7857 log.cc:1196] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/77e88228f60843a489268ddeb65a2ecc
I20260927 05:13:29.611884 7858 ts_tablet_manager.cc:1918] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:29.612025 7858 log.cc:1196] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/77e88228f60843a489268ddeb65a2ecc
I20260927 05:13:29.612469 7857 ts_tablet_manager.cc:1939] T 77e88228f60843a489268ddeb65a2ecc P 6f8a1691fbf04196a2575a07915fa7bd: Deleting consensus metadata
I20260927 05:13:29.612519 7858 ts_tablet_manager.cc:1939] T 77e88228f60843a489268ddeb65a2ecc P d42228ac01354c308bd1ff678d80d383: Deleting consensus metadata
I20260927 05:13:29.614490 6521 catalog_manager.cc:4841] TS d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501): tablet 77e88228f60843a489268ddeb65a2ecc (table test-table [id=56c8eb8a2d4742cba16f90265daadfda]) successfully deleted
I20260927 05:13:29.614490 6520 catalog_manager.cc:4841] TS 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191): tablet 77e88228f60843a489268ddeb65a2ecc (table test-table [id=56c8eb8a2d4742cba16f90265daadfda]) successfully deleted
I20260927 05:13:29.615347 6566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:29.625887 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.585573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.0.97.61:46223)
0927 05:13:29.585736 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.585743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.605122 (+ 19379us) server_negotiation.cc:407] Connection header received
0927 05:13:29.605407 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.605414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.605500 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.605576 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.608240 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.609318 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.616014 (+ 6696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.616773 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.616924 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.620554 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.620560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.620563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.620593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.623186 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.623926 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.623932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.623937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.624053 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.624450 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.624455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.624459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.624776 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.625026 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.625336 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.625544 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:13:29.639814 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.603567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.97.60:44051)
0927 05:13:29.603777 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.603783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.626027 (+ 22244us) server_negotiation.cc:407] Connection header received
0927 05:13:29.626254 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.626262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.626348 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.626422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.628171 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.629266 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.630582 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.631195 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.631389 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.633933 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.633938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.633941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.634024 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.636890 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.637534 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.637542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.637547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.637675 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.638422 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.638429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.638432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.638701 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.638933 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.639280 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.639484 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:13:29.810324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6504
I20260927 05:13:29.854310 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.838023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.97.61:46223)
0927 05:13:29.838176 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.838182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.839664 (+ 1482us) server_negotiation.cc:407] Connection header received
0927 05:13:29.839922 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.839929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.840011 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.840082 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.841961 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.842909 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.844908 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.845441 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.845572 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.848377 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.848385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.848389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.848427 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.851712 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.852398 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.852403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.852407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.852543 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.852889 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.852895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.852898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.853155 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.853405 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.853889 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.854083 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:13:29.854354 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:29.837147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.97.60:44051)
0927 05:13:29.837386 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:29.837395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:29.838356 (+ 961us) server_negotiation.cc:407] Connection header received
0927 05:13:29.838606 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:29.838615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:29.838677 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:29.838771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:29.841125 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.842202 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.843778 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:29.844182 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:29.844306 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:29.847913 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:29.847922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:29.847928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:29.847970 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:29.851496 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.852274 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.852279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.852282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.852420 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:29.852800 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:29.852805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:29.852808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:29.853110 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:29.853339 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:29.853689 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 05:13:29.854029 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 05:13:30.558189 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:30.557404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:40213)
0927 05:13:30.557838 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:30.557941 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":250,"threads_started":1}
W20260927 05:13:30.558562 7657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:30.563100 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:30.562020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:60979)
0927 05:13:30.562756 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:30.562873 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":521,"threads_started":1}
W20260927 05:13:30.563496 7799 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:30.569383 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:30.568777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:48941)
0927 05:13:30.569150 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:30.569246 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":210,"threads_started":1}
W20260927 05:13:30.569695 7508 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.080699 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06b69ab300d2440d8a05e4fac552aab8)
I20260927 05:13:31.080964 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } }
I20260927 05:13:31.081277 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06b69ab300d2440d8a05e4fac552aab8)
I20260927 05:13:31.081494 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } }
I20260927 05:13:31.082226 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801), 38b24a74000040b88f7a78ce32933c67 (127.0.97.61:46223)
I20260927 05:13:31.082465 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801), 973da5290c9e497491e50cd4f8f34d09 (127.0.97.60:44051)
I20260927 05:13:31.083539 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.082351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37520)
0927 05:13:31.083067 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.083226 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":441,"threads_started":1}
W20260927 05:13:31.085150 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.088393 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.082530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37536)
0927 05:13:31.087836 (+ 5306us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.088090 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5129,"thread_start_us":4990,"threads_started":1}
W20260927 05:13:31.088712 7120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.095364 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.094763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37540)
0927 05:13:31.094945 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.095118 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:13:31.095700 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.098207 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.097846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37546)
0927 05:13:31.097977 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.098049 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 05:13:31.098492 7120 leader_election.cc:334] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.099699 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.083177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46223 (local address 127.0.0.1:44578)
0927 05:13:31.083787 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.083810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.083962 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.093486 (+ 9524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.093492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.093519 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.093763 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.093773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.095493 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.095499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.097495 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.097509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.099158 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.099163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.099280 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.099304 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.099391 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.099479 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"spinlock_wait_cycles":5504,"thread_start_us":1642,"threads_started":1}
I20260927 05:13:31.099913 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.084252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.0.97.61:46223)
0927 05:13:31.093105 (+ 8853us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.093110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.093127 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:31.093179 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.093184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.093249 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.093895 (+ 646us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.094074 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.095321 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.097645 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.098954 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.099170 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.099264 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.099428 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.099561 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":8744,"thread_start_us":8565,"threads_started":1}
I20260927 05:13:31.100339 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973da5290c9e497491e50cd4f8f34d09" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "38b24a74000040b88f7a78ce32933c67" is_pre_election: true
I20260927 05:13:31.100576 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973da5290c9e497491e50cd4f8f34d09 in term 1.
I20260927 05:13:31.101069 7315 leader_election.cc:302] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38b24a74000040b88f7a78ce32933c67, 973da5290c9e497491e50cd4f8f34d09; no voters: 06b69ab300d2440d8a05e4fac552aab8
I20260927 05:13:31.101336 7871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 05:13:31.101459 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 06b69ab300d2440d8a05e4fac552aab8)
I20260927 05:13:31.101558 7871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:13:31.102787 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.083569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.0.97.60:44051)
0927 05:13:31.093378 (+ 9809us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.093386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.093411 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:13:31.093482 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.093491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.093604 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.093735 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.097263 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.098367 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.100846 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.101920 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.102018 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.102107 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.102432 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.102522 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9672,"spinlock_wait_cycles":1408,"thread_start_us":9485,"threads_started":1}
I20260927 05:13:31.102865 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.082631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44051 (local address 127.0.0.1:36520)
0927 05:13:31.088150 (+ 5519us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.088173 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.088323 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.096717 (+ 8394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.096720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.096740 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.097033 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.097046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.098509 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.098516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.100676 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.100703 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.102084 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.102090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.102274 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.102307 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.102389 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.102516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5359,"mutex_wait_us":22,"thread_start_us":5224,"threads_started":1}
I20260927 05:13:31.103493 7338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38b24a74000040b88f7a78ce32933c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "973da5290c9e497491e50cd4f8f34d09" is_pre_election: true
I20260927 05:13:31.104426 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } }
I20260927 05:13:31.104843 7338 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38b24a74000040b88f7a78ce32933c67 in current term 2: Already voted for candidate 973da5290c9e497491e50cd4f8f34d09 in this term.
I20260927 05:13:31.105556 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 election: Requested vote from peers 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801), 38b24a74000040b88f7a78ce32933c67 (127.0.97.61:46223)
I20260927 05:13:31.105738 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.105155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37560)
0927 05:13:31.105284 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.105351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:13:31.105813 7118 leader_election.cc:302] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38b24a74000040b88f7a78ce32933c67; no voters: 06b69ab300d2440d8a05e4fac552aab8, 973da5290c9e497491e50cd4f8f34d09
I20260927 05:13:31.105836 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973da5290c9e497491e50cd4f8f34d09" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "38b24a74000040b88f7a78ce32933c67"
I20260927 05:13:31.106109 7143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:13:31.106668 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.106338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37576)
0927 05:13:31.106490 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.106566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 05:13:31.106796 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
I20260927 05:13:31.108157 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973da5290c9e497491e50cd4f8f34d09 in term 2.
I20260927 05:13:31.108326 7870 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:13:31.108592 7315 leader_election.cc:302] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38b24a74000040b88f7a78ce32933c67, 973da5290c9e497491e50cd4f8f34d09; no voters: 06b69ab300d2440d8a05e4fac552aab8
I20260927 05:13:31.108878 7871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:13:31.109282 7871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [term 2 LEADER]: Becoming Leader. State: Replica: 973da5290c9e497491e50cd4f8f34d09, State: Running, Role: LEADER
I20260927 05:13:31.109627 7871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } }
I20260927 05:13:31.111195 7880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973da5290c9e497491e50cd4f8f34d09. Latest consensus state: current_term: 2 leader_uuid: "973da5290c9e497491e50cd4f8f34d09" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:31.111615 7880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:31.112239 7882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:31.115619 7882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:31.116396 7882 catalog_manager.cc:1259] Loaded cluster ID: da3c825fa4fe44899b55c976c03a84de
I20260927 05:13:31.116496 7882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:31.117560 7882 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:31.118343 7882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Loaded TSK: 0
I20260927 05:13:31.119177 7882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:13:31.135955 7328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:31.142534 7328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:31.146296 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.145940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37596)
0927 05:13:31.146108 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.146173 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 05:13:31.146272 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [term 2 FOLLOWER]: Refusing update from remote peer 973da5290c9e497491e50cd4f8f34d09: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 05:13:31.146970 7880 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 05:13:31.147425 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.147075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37608)
0927 05:13:31.147243 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.147315 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 05:13:31.147639 7315 consensus_peers.cc:487] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 -> Peer 06b69ab300d2440d8a05e4fac552aab8 (127.0.97.62:41801): Couldn't send request to peer 06b69ab300d2440d8a05e4fac552aab8. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:13:31.149412 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973da5290c9e497491e50cd4f8f34d09. Latest consensus state: current_term: 2 leader_uuid: "973da5290c9e497491e50cd4f8f34d09" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:31.149668 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:31.150956 7880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "973da5290c9e497491e50cd4f8f34d09" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:31.151214 7884 mvcc.cc:205] Tried to move back new op lower bound from 7333830701649358848 to 7333830701508001792. Current Snapshot: MvccSnapshot[applied={T|T < 7333830701649358848}]
I20260927 05:13:31.151274 7880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09 [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:31.158921 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "973da5290c9e497491e50cd4f8f34d09" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "06b69ab300d2440d8a05e4fac552aab8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41801 } } peers { permanent_uuid: "38b24a74000040b88f7a78ce32933c67" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46223 } attrs { promote: false } } peers { permanent_uuid: "973da5290c9e497491e50cd4f8f34d09" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44051 } attrs { promote: false } } }
I20260927 05:13:31.159207 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b24a74000040b88f7a78ce32933c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:13:31.170135 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.162896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.0.1:35592)
0927 05:13:31.163118 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.163138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.163234 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.164201 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.164206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.164232 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.164478 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.164488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.165862 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.165870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.167754 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.167767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.169445 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.169450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.169663 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.169722 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.169815 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.169918 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":25}
I20260927 05:13:31.170646 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.163051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.0.97.3:43191)
0927 05:13:31.163826 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.163830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.163844 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:13:31.163921 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.163929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.164027 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.164591 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.164724 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.165705 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.167931 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.169113 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.169230 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.169361 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.170331 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.170443 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":592,"threads_started":1}
I20260927 05:13:31.171351 7733 tablet_service.cc:1467] Processing CreateTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 (DEFAULT_TABLE table=test-table [id=5232549da64040a8aa872506c516d61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:31.171746 7733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93f23875faa4ae580270ac9fb15f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:31.172613 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.163816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.0.97.2:44501)
0927 05:13:31.166185 (+ 2369us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.166191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.166209 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:13:31.166273 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.166278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.166348 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.166449 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.167075 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.168028 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.170021 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.171495 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.171601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.171691 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.172195 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.172332 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2260,"thread_start_us":2168,"threads_started":1}
I20260927 05:13:31.172638 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.163730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44501 (local address 127.0.0.1:58860)
0927 05:13:31.164265 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.164283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.164375 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.166496 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.166504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.166539 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.166902 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.166910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.168154 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.168161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.169844 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.169858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.171852 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.171858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.172024 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.172058 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.172151 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.172276 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":434,"threads_started":1}
I20260927 05:13:31.173292 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.163235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:50004)
0927 05:13:31.163365 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.163383 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.163498 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.167239 (+ 3741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.167243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.167267 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.167614 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.167627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.168886 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.168892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.170905 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.170918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.172562 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.172568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.172767 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.172809 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.172876 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.173007 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":27}
I20260927 05:13:31.173494 7591 tablet_service.cc:1467] Processing CreateTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 (DEFAULT_TABLE table=test-table [id=5232549da64040a8aa872506c516d61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:31.173906 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93f23875faa4ae580270ac9fb15f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:31.174103 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.164037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.0.97.1:33065)
0927 05:13:31.166884 (+ 2847us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.166890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.166907 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:31.166970 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.166976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.167099 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.167199 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.167764 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.168751 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.171127 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.172341 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.173522 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.173611 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.173661 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.173788 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2734,"thread_start_us":2612,"threads_started":1}
I20260927 05:13:31.174695 7442 tablet_service.cc:1467] Processing CreateTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 (DEFAULT_TABLE table=test-table [id=5232549da64040a8aa872506c516d61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:31.175117 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93f23875faa4ae580270ac9fb15f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:31.178010 7893 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Bootstrap starting.
I20260927 05:13:31.179872 7893 tablet_bootstrap.cc:654] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:31.180245 7894 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Bootstrap starting.
I20260927 05:13:31.180904 7895 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Bootstrap starting.
I20260927 05:13:31.181941 7894 tablet_bootstrap.cc:654] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:31.182686 7895 tablet_bootstrap.cc:654] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:31.184130 7894 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: No bootstrap required, opened a new log
I20260927 05:13:31.184266 7894 ts_tablet_manager.cc:1397] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:31.184751 7895 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: No bootstrap required, opened a new log
I20260927 05:13:31.184799 7893 tablet_bootstrap.cc:492] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: No bootstrap required, opened a new log
I20260927 05:13:31.184844 7895 ts_tablet_manager.cc:1397] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:13:31.184916 7893 ts_tablet_manager.cc:1397] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 05:13:31.185053 7894 raft_consensus.cc:348] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.185271 7894 raft_consensus.cc:374] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:31.185348 7894 raft_consensus.cc:729] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42228ac01354c308bd1ff678d80d383, State: Initialized, Role: FOLLOWER
I20260927 05:13:31.185354 7895 raft_consensus.cc:348] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.185550 7895 raft_consensus.cc:374] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:31.185635 7895 raft_consensus.cc:729] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa16bb930014639b04b857b99c7e5e0, State: Initialized, Role: FOLLOWER
I20260927 05:13:31.185578 7894 consensus_queue.cc:260] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.185681 7893 raft_consensus.cc:348] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.185876 7893 raft_consensus.cc:374] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:31.185981 7893 raft_consensus.cc:729] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8a1691fbf04196a2575a07915fa7bd, State: Initialized, Role: FOLLOWER
I20260927 05:13:31.185835 7895 consensus_queue.cc:260] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.186194 7893 consensus_queue.cc:260] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.187991 7894 ts_tablet_manager.cc:1428] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 05:13:31.189949 7656 heartbeater.cc:503] Master 127.0.97.60:44051 was elected leader, sending a full tablet report...
I20260927 05:13:31.196143 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.186536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:49617)
0927 05:13:31.195608 (+ 9072us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.195781 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8874,"thread_start_us":8685,"threads_started":1}
I20260927 05:13:31.197311 7895 ts_tablet_manager.cc:1428] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260927 05:13:31.198045 7893 ts_tablet_manager.cc:1428] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260927 05:13:31.199325 7507 heartbeater.cc:503] Master 127.0.97.60:44051 was elected leader, sending a full tablet report...
I20260927 05:13:31.200706 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.200211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:37601)
0927 05:13:31.200389 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.200505 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 05:13:31.201745 7798 heartbeater.cc:503] Master 127.0.97.60:44051 was elected leader, sending a full tablet report...
I20260927 05:13:31.203408 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.197742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:39851)
0927 05:13:31.201743 (+ 4001us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.203108 (+ 1365us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3862,"thread_start_us":3744,"threads_started":1}
I20260927 05:13:31.204149 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.197180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:35885)
0927 05:13:31.202257 (+ 5077us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.203560 (+ 1303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4923,"thread_start_us":5074,"threads_started":1}
I20260927 05:13:31.204290 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.203929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:45573)
0927 05:13:31.204044 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.204122 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 05:13:31.205787 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.205461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:43453)
0927 05:13:31.205602 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.205680 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 05:13:31.347527 7899 raft_consensus.cc:482] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:31.347721 7899 raft_consensus.cc:504] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.349160 7899 leader_election.cc:288] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191), daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:31.354915 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.349434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41111 (local address 127.0.97.3:43191)
0927 05:13:31.349624 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.349631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.349936 (+ 305us) server_negotiation.cc:407] Connection header received
0927 05:13:31.350174 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.350179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.350251 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.350342 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.350826 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.351702 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.353375 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.354072 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.354205 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.354299 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.354559 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.354682 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:13:31.354939 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.349456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.97.2:41111)
0927 05:13:31.349813 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.349836 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.350023 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.350360 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.350364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.350388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.350674 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.350685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.351830 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.351837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.353223 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.353233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.354238 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.354243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.354409 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.354432 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.354537 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.354674 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260927 05:13:31.355152 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.349521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38745 (local address 127.0.97.1:33065)
0927 05:13:31.349719 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.349725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.350265 (+ 540us) server_negotiation.cc:407] Connection header received
0927 05:13:31.350453 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.350460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.350535 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.350598 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:31.351293 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.351932 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.353553 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.354257 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.354394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.354495 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.354777 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.354897 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:13:31.355214 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.349479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.97.2:38745)
0927 05:13:31.350169 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.350181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:31.350304 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:31.350777 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:31.350783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:31.350811 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:31.351138 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.351152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.352063 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.352069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.353400 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:31.353412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.354461 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.354467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:31.354636 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.354658 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:13:31.354753 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 05:13:31.354863 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":414,"threads_started":1}
I20260927 05:13:31.355886 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93f23875faa4ae580270ac9fb15f3f6" candidate_uuid: "d42228ac01354c308bd1ff678d80d383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a1691fbf04196a2575a07915fa7bd" is_pre_election: true
I20260927 05:13:31.356148 7753 raft_consensus.cc:2454] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42228ac01354c308bd1ff678d80d383 in term 0.
I20260927 05:13:31.356029 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93f23875faa4ae580270ac9fb15f3f6" candidate_uuid: "d42228ac01354c308bd1ff678d80d383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa16bb930014639b04b857b99c7e5e0" is_pre_election: true
I20260927 05:13:31.356423 7462 raft_consensus.cc:2454] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42228ac01354c308bd1ff678d80d383 in term 0.
I20260927 05:13:31.356652 7544 leader_election.cc:302] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8a1691fbf04196a2575a07915fa7bd, d42228ac01354c308bd1ff678d80d383; no voters:
I20260927 05:13:31.357049 7899 raft_consensus.cc:2781] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:31.357275 7899 raft_consensus.cc:482] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:31.357390 7899 raft_consensus.cc:3037] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:31.359547 7899 raft_consensus.cc:504] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.360075 7899 leader_election.cc:288] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [CANDIDATE]: Term 1 election: Requested vote from peers 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191), daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065)
I20260927 05:13:31.360545 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93f23875faa4ae580270ac9fb15f3f6" candidate_uuid: "d42228ac01354c308bd1ff678d80d383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa16bb930014639b04b857b99c7e5e0"
I20260927 05:13:31.360545 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93f23875faa4ae580270ac9fb15f3f6" candidate_uuid: "d42228ac01354c308bd1ff678d80d383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a1691fbf04196a2575a07915fa7bd"
I20260927 05:13:31.360735 7753 raft_consensus.cc:3037] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:31.360735 7462 raft_consensus.cc:3037] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:31.363723 7753 raft_consensus.cc:2454] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42228ac01354c308bd1ff678d80d383 in term 1.
I20260927 05:13:31.363723 7462 raft_consensus.cc:2454] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42228ac01354c308bd1ff678d80d383 in term 1.
I20260927 05:13:31.364128 7546 leader_election.cc:302] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d42228ac01354c308bd1ff678d80d383, daa16bb930014639b04b857b99c7e5e0; no voters:
I20260927 05:13:31.364406 7899 raft_consensus.cc:2781] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:31.364815 7899 raft_consensus.cc:686] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 1 LEADER]: Becoming Leader. State: Replica: d42228ac01354c308bd1ff678d80d383, State: Running, Role: LEADER
I20260927 05:13:31.365486 7899 consensus_queue.cc:237] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } }
I20260927 05:13:31.365725 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.365358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:48043)
0927 05:13:31.365490 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.365591 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 05:13:31.368641 7328 catalog_manager.cc:5495] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 reported cstate change: term changed from 0 to 1, leader changed from <none> to d42228ac01354c308bd1ff678d80d383 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d42228ac01354c308bd1ff678d80d383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42228ac01354c308bd1ff678d80d383" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44501 } health_report { overall_health: HEALTHY } } }
W20260927 05:13:31.386584 7510 tablet.cc:2369] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:31.407407 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.402604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.97.2:44501)
0927 05:13:31.402778 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.402782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.402853 (+ 71us) server_negotiation.cc:407] Connection header received
0927 05:13:31.403067 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.403071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.403152 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.403278 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:31.403572 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.404245 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.405533 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.405999 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.406100 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.406419 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:31.406926 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:31.407095 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.407140 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.407197 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:13:31.411451 7753 raft_consensus.cc:1261] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Refusing update from remote peer d42228ac01354c308bd1ff678d80d383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:31.411697 7462 raft_consensus.cc:1261] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Refusing update from remote peer d42228ac01354c308bd1ff678d80d383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:31.412135 7906 consensus_queue.cc:1035] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8a1691fbf04196a2575a07915fa7bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:31.412521 7899 consensus_queue.cc:1035] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa16bb930014639b04b857b99c7e5e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:31.414896 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.414137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:43723)
0927 05:13:31.414593 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.414706 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260927 05:13:31.415056 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.414463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:42491)
0927 05:13:31.414677 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.414827 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:13:31.418903 7909 mvcc.cc:205] Tried to move back new op lower bound from 7333830702734655488 to 7333830702556905472. Current Snapshot: MvccSnapshot[applied={T|T < 7333830702734655488}]
I20260927 05:13:31.420372 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.419810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.1:40505)
0927 05:13:31.420090 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.420196 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 05:13:31.421916 7911 mvcc.cc:205] Tried to move back new op lower bound from 7333830702734655488 to 7333830702556905472. Current Snapshot: MvccSnapshot[applied={T|T < 7333830702734655488}]
I20260927 05:13:31.426415 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.425902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:41655)
0927 05:13:31.426188 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.426261 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 05:13:31.433559 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.432976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.3:40111)
0927 05:13:31.433246 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.433352 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 05:13:31.439220 7328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:31.439512 7328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:31.443938 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.443487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.97.2:41919)
0927 05:13:31.443666 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.443755 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:13:31.446626 7328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 973da5290c9e497491e50cd4f8f34d09: Sending DeleteTablet for 3 replicas of tablet b93f23875faa4ae580270ac9fb15f3f6
I20260927 05:13:31.447677 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:31 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:58860
I20260927 05:13:31.447765 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:31 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:58860
I20260927 05:13:31.447805 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:31 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50004
I20260927 05:13:31.448395 7733 tablet_service.cc:1514] Processing DeleteTablet for tablet b93f23875faa4ae580270ac9fb15f3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:31 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35592
I20260927 05:13:31.449400 7920 tablet_replica.cc:330] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: stopping tablet replica
I20260927 05:13:31.449592 7920 raft_consensus.cc:2229] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:31.449867 7920 raft_consensus.cc:2258] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:31.449921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 7306
I20260927 05:13:31.457172 7921 tablet_replica.cc:330] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: stopping tablet replica
I20260927 05:13:31.457402 7921 raft_consensus.cc:2229] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:31.457719 7921 raft_consensus.cc:2258] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:31.458179 7920 ts_tablet_manager.cc:1905] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:31.459127 7921 ts_tablet_manager.cc:1905] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:31.467211 7920 ts_tablet_manager.cc:1918] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:31.467381 7920 log.cc:1196] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/b93f23875faa4ae580270ac9fb15f3f6
I20260927 05:13:31.467813 7919 tablet_replica.cc:330] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: stopping tablet replica
I20260927 05:13:31.467833 7920 ts_tablet_manager.cc:1939] T b93f23875faa4ae580270ac9fb15f3f6 P daa16bb930014639b04b857b99c7e5e0: Deleting consensus metadata
I20260927 05:13:31.468094 7919 raft_consensus.cc:2229] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:31.468673 7919 raft_consensus.cc:2258] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:31.468765 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.450688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.1:33065)
0927 05:13:31.450838 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.450844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.451418 (+ 574us) server_negotiation.cc:407] Connection header received
0927 05:13:31.451677 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.451683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.451763 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.451831 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:31.454045 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.455185 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.456953 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.457643 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.457750 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.461174 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:31.461187 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:31.461205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:31.461242 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:31.465347 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.466198 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.466207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.466211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.466358 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.466817 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.466824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.466827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.467304 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:31.467640 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.468195 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.468395 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:13:31.468779 7314 catalog_manager.cc:4841] TS daa16bb930014639b04b857b99c7e5e0 (127.0.97.1:33065): tablet b93f23875faa4ae580270ac9fb15f3f6 (table test-table [id=5232549da64040a8aa872506c516d61d]) successfully deleted
I20260927 05:13:31.469455 7502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:31.470400 7919 ts_tablet_manager.cc:1905] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:31.471477 7921 ts_tablet_manager.cc:1918] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:31.471603 7921 log.cc:1196] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/b93f23875faa4ae580270ac9fb15f3f6
I20260927 05:13:31.471989 7921 ts_tablet_manager.cc:1939] T b93f23875faa4ae580270ac9fb15f3f6 P 6f8a1691fbf04196a2575a07915fa7bd: Deleting consensus metadata
I20260927 05:13:31.472852 7312 catalog_manager.cc:4841] TS 6f8a1691fbf04196a2575a07915fa7bd (127.0.97.3:43191): tablet b93f23875faa4ae580270ac9fb15f3f6 (table test-table [id=5232549da64040a8aa872506c516d61d]) successfully deleted
I20260927 05:13:31.478456 7919 ts_tablet_manager.cc:1918] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:31.478562 7919 log.cc:1196] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/b93f23875faa4ae580270ac9fb15f3f6
I20260927 05:13:31.478895 7919 ts_tablet_manager.cc:1939] T b93f23875faa4ae580270ac9fb15f3f6 P d42228ac01354c308bd1ff678d80d383: Deleting consensus metadata
I20260927 05:13:31.479882 7313 catalog_manager.cc:4841] TS d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501): tablet b93f23875faa4ae580270ac9fb15f3f6 (table test-table [id=5232549da64040a8aa872506c516d61d]) successfully deleted
W20260927 05:13:31.480293 7313 catalog_manager.cc:4808] TS d42228ac01354c308bd1ff678d80d383 (127.0.97.2:44501): delete failed for tablet b93f23875faa4ae580270ac9fb15f3f6 because the tablet was not found. No further retry: Not found: Tablet not found: b93f23875faa4ae580270ac9fb15f3f6
I20260927 05:13:31.621308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 7306
I20260927 05:13:31.642908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 7519
I20260927 05:13:31.654598 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.654209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37618)
0927 05:13:31.654362 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.654448 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 05:13:31.656131 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.655731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41801 (local address 127.0.0.1:37630)
0927 05:13:31.655886 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:31.655984 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:13:31.657018 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.643666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.97.2:44501)
0927 05:13:31.643851 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.643857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.643967 (+ 110us) server_negotiation.cc:407] Connection header received
0927 05:13:31.644149 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.644154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.644226 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.644337 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:31.645545 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.646378 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.647820 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.648239 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.648372 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.650720 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:31.650737 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:31.650753 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:31.650792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:31.654021 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.654812 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.654823 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.654831 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.655040 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.655483 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.655488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.655491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.655886 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:31.656324 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.656546 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.656707 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:13:31.657652 7651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:31.788175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 7519
I20260927 05:13:31.808236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 7669
I20260927 05:13:31.821765 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:31.808906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.0.97.3:43191)
0927 05:13:31.809098 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:31.809105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:31.809126 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:13:31.809313 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:31.809318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:31.809388 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:31.809486 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:31.810776 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.811646 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.812912 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:31.813289 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:31.813418 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:31.815919 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:31.815932 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:31.815949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:31.815983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:31.818476 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.819472 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.819481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.819485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.819627 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:31.820172 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:31.820180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:31.820184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:31.820565 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:31.820898 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:31.821218 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 05:13:31.821472 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:13:31.822484 7793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:31.918494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 7669
I20260927 05:13:31.935060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 6997
I20260927 05:13:31.936074 7163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:32.048547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 6997
I20260927 05:13:32.064775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 7097
I20260927 05:13:32.065807 7358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:32.183548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 7097
2026-09-27T05:13:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9244 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:32 dist-test-slave-1t6p krb5kdc[7931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:32 dist-test-slave-1t6p krb5kdc[7931](info): set up 2 sockets
Sep 27 05:13:32 dist-test-slave-1t6p krb5kdc[7931](info): commencing operation
krb5kdc: starting...
W20260927 05:13:33.263765 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:33 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 05:13:33.456151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:33.768080 7944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:33.768339 7944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:33.768427 7944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:33.775279 7944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:33.775365 7944 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:13:33.775404 7944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:33.775457 7944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:33.775497 7944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:33.775554 7944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:33.784651 7944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40995
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:33.785722 7944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:33.786916 7944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:33.798432 7952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:33.798519 7950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:33.798395 7949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:33.799382 7944 server_base.cc:1034] running on GCE node
I20260927 05:13:33.800146 7944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:33.801355 7944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:33.802590 7944 hybrid_clock.cc:648] HybridClock initialized: now 1790486013802532 us; error 51 us; skew 500 ppm
I20260927 05:13:33.802891 7944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:33 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486013, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:33.805728 7944 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 05:13:33.807035 7944 webserver.cc:460] Webserver started at http://127.0.97.62:45679/ using document root <none> and password file <none>
I20260927 05:13:33.807685 7944 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:33.807806 7944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:33.808108 7944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:33.810940 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "1975cfa807e244779bf40c2beee6b26f"
format_stamp: "Formatted at 2026-09-27 05:13:33 on dist-test-slave-1t6p"
server_key: "451ea9bd9ecd58ff0fc5aa3bfbb0f0d6"
server_key_iv: "98b6e98b0d8ac1126a3a7b1d7b26987b"
server_key_version: "encryptionkey@0"
I20260927 05:13:33.811743 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "1975cfa807e244779bf40c2beee6b26f"
format_stamp: "Formatted at 2026-09-27 05:13:33 on dist-test-slave-1t6p"
server_key: "451ea9bd9ecd58ff0fc5aa3bfbb0f0d6"
server_key_iv: "98b6e98b0d8ac1126a3a7b1d7b26987b"
server_key_version: "encryptionkey@0"
I20260927 05:13:33.816308 7944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 05:13:33.819468 7959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:33.820677 7944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:13:33.820844 7944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "1975cfa807e244779bf40c2beee6b26f"
format_stamp: "Formatted at 2026-09-27 05:13:33 on dist-test-slave-1t6p"
server_key: "451ea9bd9ecd58ff0fc5aa3bfbb0f0d6"
server_key_iv: "98b6e98b0d8ac1126a3a7b1d7b26987b"
server_key_version: "encryptionkey@0"
I20260927 05:13:33.821043 7944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:33.837841 7944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:33.841563 7944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:33.841845 7944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:33.864526 7944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40995
I20260927 05:13:33.864545 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40995 every 8 connection(s)
I20260927 05:13:33.865908 7944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:33.869457 8021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:33.872820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 7944
I20260927 05:13:33.873086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:33.873674 388 external_mini_cluster.cc:1428] Setting key 6f348397b4e772d525ef8011d19adafc
I20260927 05:13:33.882145 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Bootstrap starting.
I20260927 05:13:33.885690 8021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:33.886647 8021 log.cc:824] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:13:33 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:33.890089 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: No bootstrap required, opened a new log
I20260927 05:13:33.896715 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:33.875595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.62:40995)
0927 05:13:33.876331 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:33.876345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:33.876392 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:13:33.878040 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:33.878071 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:33.878539 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:33.878947 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:33.880198 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:33.882141 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:33.883464 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:33.884547 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:33.884722 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:33.888609 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:33.888616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:33.888632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:33.888665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:33.892306 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:33.892960 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:33.892968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:33.892977 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:33.893185 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:33.893515 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:33.893520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:33.893523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:33.894202 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:33.894510 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:33.894869 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 05:13:33.895407 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":305,"threads_started":1}
I20260927 05:13:33.900111 8021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } }
I20260927 05:13:33.900410 8021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:33.900470 8021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1975cfa807e244779bf40c2beee6b26f, State: Initialized, Role: FOLLOWER
I20260927 05:13:33.900920 8021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } }
I20260927 05:13:33.901106 8021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:33.901201 8021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:33.901311 8021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:33.903293 8021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } }
I20260927 05:13:33.903709 8021 leader_election.cc:302] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1975cfa807e244779bf40c2beee6b26f; no voters:
I20260927 05:13:33.904294 8021 leader_election.cc:288] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:33.904502 8026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:33.905522 8026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [term 1 LEADER]: Becoming Leader. State: Replica: 1975cfa807e244779bf40c2beee6b26f, State: Running, Role: LEADER
I20260927 05:13:33.906011 8026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } }
I20260927 05:13:33.906316 8021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:33.908591 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1975cfa807e244779bf40c2beee6b26f. Latest consensus state: current_term: 1 leader_uuid: "1975cfa807e244779bf40c2beee6b26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } } }
I20260927 05:13:33.908619 8027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1975cfa807e244779bf40c2beee6b26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975cfa807e244779bf40c2beee6b26f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40995 } } }
I20260927 05:13:33.909031 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:33.909032 8027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:33.909899 8031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:33.915156 8031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:33.922873 8031 catalog_manager.cc:1347] Generated new cluster ID: 7afb5b1bcf064da180b200b8f46ebb65
I20260927 05:13:33.922981 8031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:33.953118 8031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:33.954381 8031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:33.963763 8031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Generated new TSK 0
I20260927 05:13:33.964558 8031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 05:13:34.022760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40995
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:34.342478 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:34.342770 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:34.342903 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:34.349869 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:34.349970 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:34.350050 8047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:13:34.350133 8047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:34.359405 8047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:34.360534 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:34.361742 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:34.374133 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:34.374171 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:34.374199 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:34.375344 8047 server_base.cc:1034] running on GCE node
I20260927 05:13:34.375926 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:34.376802 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:34.378041 8047 hybrid_clock.cc:648] HybridClock initialized: now 1790486014377975 us; error 36 us; skew 500 ppm
I20260927 05:13:34.378317 8047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:34 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:34.381361 8047 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:13:34.382671 8047 webserver.cc:460] Webserver started at http://127.0.97.1:35437/ using document root <none> and password file <none>
I20260927 05:13:34.383399 8047 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:34.383519 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:34.383824 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:34.386529 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "44729f2a086e4455b0db34f7d3404817"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "821190688fb0027799573ec7f0252632"
server_key_iv: "a2ee1a26900280edaf0558ed0089a234"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.387342 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "44729f2a086e4455b0db34f7d3404817"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "821190688fb0027799573ec7f0252632"
server_key_iv: "a2ee1a26900280edaf0558ed0089a234"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.392179 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 05:13:34.395617 8062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:34.396991 8047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:13:34.397156 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "44729f2a086e4455b0db34f7d3404817"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "821190688fb0027799573ec7f0252632"
server_key_iv: "a2ee1a26900280edaf0558ed0089a234"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.397361 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:34.417754 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:34.421384 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:34.421640 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:34.424309 8047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:34.424408 8047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:34.424551 8047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:34.424675 8047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 05:13:34 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:34.447237 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.424775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40227 (local address 127.0.97.62:40995)
0927 05:13:34.425630 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:34.425634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:34.426292 (+ 658us) server_negotiation.cc:407] Connection header received
0927 05:13:34.428296 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:34.428301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:34.428383 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:34.428503 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:34.430413 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.431402 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.434640 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.435096 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.435233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.440404 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:34.440414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:34.440419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:34.440469 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:34.444120 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.445039 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.445046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.445052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.445180 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.445598 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.445604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.445611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.445872 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:34.446027 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:34.446639 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 05:13:34.446838 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":656,"threads_started":1}
I20260927 05:13:34.448256 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.425420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.1:40227)
0927 05:13:34.426147 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:34.426180 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:34.428031 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:34.428745 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:34.428759 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:34.429440 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:34.430243 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.430265 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.431591 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.431603 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.434502 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.434516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.435523 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.435529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.435815 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.436596 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:34.436644 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:34.440093 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:34.444247 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.444255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.444273 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.444903 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.445290 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.445295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.445299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.445490 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.446009 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:34.446016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:34.446322 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 05:13:34.446571 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:13:34.447073 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":313,"threads_started":1}
I20260927 05:13:34.473973 8047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37673
I20260927 05:13:34.474012 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37673 every 8 connection(s)
I20260927 05:13:34.475419 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:34.477774 8183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40995
I20260927 05:13:34.478325 8183 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:34.479140 8183 heartbeater.cc:511] Master 127.0.97.62:40995 requested a full tablet report, sending...
I20260927 05:13:34.481276 7976 ts_manager.cc:194] Registered new tserver with Master: 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673)
I20260927 05:13:34.483147 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.469660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37695 (local address 127.0.97.62:40995)
0927 05:13:34.469860 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:34.469869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:34.470515 (+ 646us) server_negotiation.cc:407] Connection header received
0927 05:13:34.470725 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:34.470729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:34.470788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:34.470860 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:34.472339 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.473129 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.474631 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.475298 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.475441 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.477869 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:34.477874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:34.477878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:34.477916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:34.480704 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.481429 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.481433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.481435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.481551 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.481909 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.481912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.481914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.482172 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:34.482277 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:34.482705 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:13:34.482852 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:13:34.483330 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.469638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.1:37695)
0927 05:13:34.470390 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:34.470415 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:34.470588 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:34.470921 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:34.470929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:34.471765 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:34.472188 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.472200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.473279 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.473284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.474464 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.474476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.475469 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.475490 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.475649 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.476358 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:34.476383 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:34.477639 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:34.480824 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.480830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.480835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.481314 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.481649 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.481653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.481656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.481813 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.482284 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:34.482289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:34.482474 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 05:13:34.482621 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 05:13:34.482894 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":391,"threads_started":1}
I20260927 05:13:34.483335 7976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40227
I20260927 05:13:34.485015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8047
I20260927 05:13:34.485226 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:34.486006 388 external_mini_cluster.cc:1428] Setting key a83bba42a59a285db37d14edda0f0c18
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 05:13:34.531760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40995
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:34.836990 8187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:34.837255 8187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:34.837347 8187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:34.844317 8187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:34.844411 8187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:34.844534 8187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:13:34.844671 8187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:34.854285 8187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:34.855444 8187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:34.856706 8187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:34.869104 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:34.869076 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:34.869805 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:34.870590 8187 server_base.cc:1034] running on GCE node
I20260927 05:13:34.871241 8187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:34.872210 8187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:34.873476 8187 hybrid_clock.cc:648] HybridClock initialized: now 1790486014873400 us; error 48 us; skew 500 ppm
I20260927 05:13:34.873780 8187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:34 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:34.877017 8187 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:13:34.878531 8187 webserver.cc:460] Webserver started at http://127.0.97.2:34505/ using document root <none> and password file <none>
I20260927 05:13:34.879442 8187 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:34.879590 8187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:34.879911 8187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:34.882864 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "1be4a0567a984545990f322e7b5be382"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "57a881a9e8561f8f6d3e859276ad0b43"
server_key_iv: "87d969a6ab274be7d81eb27a9f8db391"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.883718 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1be4a0567a984545990f322e7b5be382"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "57a881a9e8561f8f6d3e859276ad0b43"
server_key_iv: "87d969a6ab274be7d81eb27a9f8db391"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.888585 8187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:13:34.891896 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:34.893198 8187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:13:34.893373 8187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "1be4a0567a984545990f322e7b5be382"
format_stamp: "Formatted at 2026-09-27 05:13:34 on dist-test-slave-1t6p"
server_key: "57a881a9e8561f8f6d3e859276ad0b43"
server_key_iv: "87d969a6ab274be7d81eb27a9f8db391"
server_key_version: "encryptionkey@0"
I20260927 05:13:34.893608 8187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:34.911995 8187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:34.915619 8187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:34.915860 8187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:34.918287 8187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:34.918392 8187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:34.918547 8187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:34.918610 8187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:13:34 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:34.941306 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.918627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47963 (local address 127.0.97.62:40995)
0927 05:13:34.918809 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:34.918815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:34.920190 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 05:13:34.922163 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:34.922172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:34.922263 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:34.922406 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:34.924569 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.925998 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.928347 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.929069 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.929218 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.933958 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:34.933968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:34.933976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:34.934023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:34.937518 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.938642 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.938649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.938655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.938811 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.939333 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.939338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.939342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.939679 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:34.939826 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:34.940760 (+ 934us) server_negotiation.cc:299] Negotiation successful
0927 05:13:34.940976 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:13:34.942269 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.919168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.2:47963)
0927 05:13:34.920010 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:34.920062 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:34.921901 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:34.922666 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:34.922678 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:34.923479 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:34.924365 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.924401 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.926199 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.926205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.928192 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.928204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.929546 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.929557 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.929814 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.930652 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:34.930684 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:34.933747 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:34.937707 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.937718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.937736 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.938497 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.938913 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.938923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.938927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.939219 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.939897 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:34.939903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:34.940355 (+ 452us) client_negotiation.cc:769] Sending connection context
0927 05:13:34.940663 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 05:13:34.941149 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260927 05:13:34.963959 8187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42023
I20260927 05:13:34.963986 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42023 every 8 connection(s)
I20260927 05:13:34.965196 8187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:34.967321 8322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40995
I20260927 05:13:34.967581 8322 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:34.968375 8322 heartbeater.cc:511] Master 127.0.97.62:40995 requested a full tablet report, sending...
I20260927 05:13:34.969837 7975 ts_manager.cc:194] Registered new tserver with Master: 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023)
I20260927 05:13:34.969858 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.949810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57899 (local address 127.0.97.62:40995)
0927 05:13:34.950071 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:34.950079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:34.954460 (+ 4381us) server_negotiation.cc:407] Connection header received
0927 05:13:34.954598 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:34.954603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:34.954681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:34.954772 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:34.956094 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.957304 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.959996 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.960749 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.960896 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.963598 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:34.963606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:34.963612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:34.963652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:34.966944 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.967913 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.967917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.967919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.968024 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:34.968393 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:34.968396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:34.968398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:34.968723 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:34.968830 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:34.969295 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 05:13:34.969499 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 05:13:34.969909 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:34.949315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.2:57899)
0927 05:13:34.954348 (+ 5033us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:34.954364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:34.954488 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:34.954830 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:34.954834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:34.955479 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:34.955912 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.955931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.957479 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.957488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.959833 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:34.959844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:34.960980 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:34.960989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:34.961255 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:34.961998 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:34.962029 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:34.963344 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:34.967164 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.967169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.967173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.967795 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.968128 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:34.968132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:34.968136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:34.968293 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:34.968832 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:34.968837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:34.969043 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 05:13:34.969193 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 05:13:34.969489 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4904,"thread_start_us":4798,"threads_started":1}
I20260927 05:13:34.969973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8187
I20260927 05:13:34.970149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:34.970664 388 external_mini_cluster.cc:1428] Setting key 7d82ab83c27c35a54714afb85c872169
I20260927 05:13:34.970986 7975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47963
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:13:35.014564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40995
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:35.335685 8326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:35.335968 8326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:35.336135 8326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:35.342896 8326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:35.343107 8326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:35.343255 8326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:13:35.343386 8326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:35.352749 8326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:35.353861 8326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:35.355127 8326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:35.367123 8331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:35.367364 8334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:35.367416 8332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:35.368265 8326 server_base.cc:1034] running on GCE node
I20260927 05:13:35.368769 8326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:35.369529 8326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:35.370761 8326 hybrid_clock.cc:648] HybridClock initialized: now 1790486015370702 us; error 40 us; skew 500 ppm
I20260927 05:13:35.371093 8326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:35.373989 8326 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:13:35.375345 8326 webserver.cc:460] Webserver started at http://127.0.97.3:42371/ using document root <none> and password file <none>
I20260927 05:13:35.375922 8326 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:35.376003 8326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:35.376250 8326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:35.378844 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "81dd4404cf41406bb590fbe8895c95e5"
format_stamp: "Formatted at 2026-09-27 05:13:35 on dist-test-slave-1t6p"
server_key: "b170b2148df7323bbb19c7b65d916ae6"
server_key_iv: "6c7b4d79dd3548382f2d50eaa03eeafb"
server_key_version: "encryptionkey@0"
I20260927 05:13:35.379623 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "81dd4404cf41406bb590fbe8895c95e5"
format_stamp: "Formatted at 2026-09-27 05:13:35 on dist-test-slave-1t6p"
server_key: "b170b2148df7323bbb19c7b65d916ae6"
server_key_iv: "6c7b4d79dd3548382f2d50eaa03eeafb"
server_key_version: "encryptionkey@0"
I20260927 05:13:35.384189 8326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:35.387346 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:35.388535 8326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:13:35.388695 8326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "81dd4404cf41406bb590fbe8895c95e5"
format_stamp: "Formatted at 2026-09-27 05:13:35 on dist-test-slave-1t6p"
server_key: "b170b2148df7323bbb19c7b65d916ae6"
server_key_iv: "6c7b4d79dd3548382f2d50eaa03eeafb"
server_key_version: "encryptionkey@0"
I20260927 05:13:35.388865 8326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:35.401417 8326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:35.405098 8326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:35.405334 8326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:35.407964 8326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:35.408068 8326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:35.408257 8326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:35.408378 8326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:35.428424 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.408260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55595 (local address 127.0.97.62:40995)
0927 05:13:35.408438 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.408454 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.409770 (+ 1316us) server_negotiation.cc:407] Connection header received
0927 05:13:35.411949 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.411955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.412050 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.412226 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:35.414415 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.415540 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.417096 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.417518 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.417640 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.421577 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:35.421585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:35.421592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:35.421633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:35.424812 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.425730 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.425738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.425744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.425962 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.426368 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.426376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.426382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.426793 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:35.427028 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.427854 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.428090 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:13:35.429236 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.408777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.3:55595)
0927 05:13:35.409611 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.409658 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.411658 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.412474 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.412484 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.413295 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:35.414164 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.414193 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.415692 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.415697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.416929 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.416940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.417731 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.417738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.417927 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.418516 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:35.418535 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:35.421309 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:35.424932 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:35.424942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:35.424957 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:35.425603 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:35.426095 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:35.426099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:35.426101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:35.426265 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:35.426957 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:35.426964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:35.427451 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.427714 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.428135 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":360,"threads_started":1}
I20260927 05:13:35.446480 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.434553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36437 (local address 127.0.97.62:40995)
0927 05:13:35.434681 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.434685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.435375 (+ 690us) server_negotiation.cc:407] Connection header received
0927 05:13:35.435540 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.435549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.435599 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.435683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:35.436462 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.437103 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.438367 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.438868 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.438994 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.441151 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:35.441155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:35.441158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:35.441210 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:35.444112 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.444726 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.444731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.444734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.444853 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.445168 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.445173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.445177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.445430 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:35.445571 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.445958 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.446149 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 05:13:35.446595 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.434646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.97.3:36437)
0927 05:13:35.435289 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.435303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.435430 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.435702 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.435706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.436084 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:35.436333 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.436340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.437225 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.437233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.438217 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.438243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.439150 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.439159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.439436 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.440123 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:35.440151 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:35.440964 (+ 813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:35.444243 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:35.444262 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:35.444267 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:35.444612 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:35.444958 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:35.444967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:35.444969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:35.445070 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:35.445541 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:35.445545 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:35.445698 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.445824 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.446137 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":396,"threads_started":1}
I20260927 05:13:35.464975 8326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34897
I20260927 05:13:35.465010 8460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34897 every 8 connection(s)
I20260927 05:13:35.466173 8326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:35.468143 8461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40995
I20260927 05:13:35.468359 8461 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:35.468937 8461 heartbeater.cc:511] Master 127.0.97.62:40995 requested a full tablet report, sending...
I20260927 05:13:35.470183 7976 ts_manager.cc:194] Registered new tserver with Master: 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:35.470991 7976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55595
I20260927 05:13:35.471941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8326
I20260927 05:13:35.472121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:35.472512 388 external_mini_cluster.cc:1428] Setting key 9b5a983ea7dd18119133ed9c77bb40cc
I20260927 05:13:35.475757 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:13:35.486642 8183 heartbeater.cc:503] Master 127.0.97.62:40995 was elected leader, sending a full tablet report...
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:35.523258 8071 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:35.513489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.97.62:40995)
0927 05:13:35.513654 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.513659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.513977 (+ 318us) server_negotiation.cc:407] Connection header received
0927 05:13:35.514135 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.514140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.514201 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.514277 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:35.515582 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.516318 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.517570 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.518004 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.518121 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.523042 (+ 4921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 05:13:35 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:35.538334 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.526162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.0.97.62:40995)
0927 05:13:35.526355 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.526362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.526604 (+ 242us) server_negotiation.cc:407] Connection header received
0927 05:13:35.526778 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.526783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.526856 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.526949 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:35.528020 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.528679 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.529812 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.530221 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.530355 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.533672 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:35.533677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:35.533680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:35.533708 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:35.536169 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.536851 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.536854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.536857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.536973 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:35.537279 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:35.537282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:35.537284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:35.537527 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:35.537716 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.537871 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.538020 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:13:35.541884 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:35.548404 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:35.594379 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.560235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.97.1:37673)
0927 05:13:35.585674 (+ 25439us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.585682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.587042 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 05:13:35.587336 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.587342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.587544 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.587694 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:35.588402 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589619 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.591992 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.593192 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.593318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.593541 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.593884 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.594065 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25268,"thread_start_us":25436,"threads_started":1}
I20260927 05:13:35.594884 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.559370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.0.1:35220)
0927 05:13:35.586265 (+ 26895us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.586282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.586406 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.587279 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.587285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.587319 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:35.587780 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.587793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589146 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.589154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.591736 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.591749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.594011 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.594019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.594175 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.594203 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.594273 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.594401 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26786,"mutex_wait_us":78,"thread_start_us":27272,"threads_started":1}
I20260927 05:13:35.595166 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.559789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37673 (local address 127.0.0.1:34086)
0927 05:13:35.586892 (+ 27103us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.586910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.587153 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.587835 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.587839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.587864 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:35.588215 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.588226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589795 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.589800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.591852 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.591866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.593374 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.593380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.593621 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.593691 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.594661 (+ 970us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.594775 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26995,"thread_start_us":24887,"threads_started":1}
I20260927 05:13:35.595175 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.560033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.97.2:42023)
0927 05:13:35.584499 (+ 24466us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.584511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.586528 (+ 2017us) server_negotiation.cc:407] Connection header received
0927 05:13:35.586629 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.586636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.586854 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.587072 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:35.587946 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589023 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.591878 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.593282 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.593405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.594595 (+ 1190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.594694 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.594857 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24012,"thread_start_us":24090,"threads_started":1}
I20260927 05:13:35.595187 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.559479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.97.3:34897)
0927 05:13:35.585844 (+ 26365us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.585852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.585898 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:13:35.586024 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.586033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.586306 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.586512 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:35.587620 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589155 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.592182 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.593535 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.593655 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.593842 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.594709 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.594863 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26223,"thread_start_us":26820,"threads_started":1}
I20260927 05:13:35.595434 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.559221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34897 (local address 127.0.0.1:56578)
0927 05:13:35.585046 (+ 25825us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.585081 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.585252 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.586833 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.586841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.586880 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:35.587347 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.587362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.589300 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.589310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.592033 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.592048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.593711 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.593717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.593883 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.593914 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.593985 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.594156 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25573,"mutex_wait_us":143,"thread_start_us":26798,"threads_started":1}
I20260927 05:13:35.598899 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d (DEFAULT_TABLE table=test-table [id=0e5293643db742218c0292b236a3a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.599663 8252 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d (DEFAULT_TABLE table=test-table [id=0e5293643db742218c0292b236a3a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.600266 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0f2d63ae9b464cad5fa3397a68541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.600739 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0f2d63ae9b464cad5fa3397a68541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.607897 8113 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d (DEFAULT_TABLE table=test-table [id=0e5293643db742218c0292b236a3a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.609092 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0f2d63ae9b464cad5fa3397a68541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.610947 8485 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Bootstrap starting.
I20260927 05:13:35.612777 8486 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Bootstrap starting.
I20260927 05:13:35.614506 8485 tablet_bootstrap.cc:654] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.616216 8486 tablet_bootstrap.cc:654] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.616276 8485 log.cc:824] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:35.617509 8486 log.cc:824] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:35.620586 8489 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Bootstrap starting.
I20260927 05:13:35.621490 8486 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: No bootstrap required, opened a new log
I20260927 05:13:35.621584 8485 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: No bootstrap required, opened a new log
I20260927 05:13:35.621749 8486 ts_tablet_manager.cc:1397] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 05:13:35.621819 8485 ts_tablet_manager.cc:1397] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260927 05:13:35.624059 8489 tablet_bootstrap.cc:654] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.625578 8489 log.cc:824] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:35.629220 8486 raft_consensus.cc:348] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.629552 8486 raft_consensus.cc:374] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.629702 8486 raft_consensus.cc:729] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.630321 8486 consensus_queue.cc:260] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.631392 8461 heartbeater.cc:503] Master 127.0.97.62:40995 was elected leader, sending a full tablet report...
I20260927 05:13:35.631922 8486 ts_tablet_manager.cc:1428] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260927 05:13:35.632198 8489 tablet_bootstrap.cc:492] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: No bootstrap required, opened a new log
I20260927 05:13:35.632507 8489 ts_tablet_manager.cc:1397] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260927 05:13:35.633558 8485 raft_consensus.cc:348] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.633849 8485 raft_consensus.cc:374] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.633952 8485 raft_consensus.cc:729] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.634590 8485 consensus_queue.cc:260] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.635653 8322 heartbeater.cc:503] Master 127.0.97.62:40995 was elected leader, sending a full tablet report...
I20260927 05:13:35.636256 8485 ts_tablet_manager.cc:1428] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260927 05:13:35.641757 8489 raft_consensus.cc:348] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.642021 8489 raft_consensus.cc:374] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.642086 8489 raft_consensus.cc:729] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.642457 8489 consensus_queue.cc:260] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.643570 8489 ts_tablet_manager.cc:1428] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260927 05:13:35.717813 8323 tablet.cc:2369] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:35.717813 8462 tablet.cc:2369] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:13:35.728338 8184 tablet.cc:2369] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:13:35.799046 8491 raft_consensus.cc:482] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:35.799338 8491 raft_consensus.cc:504] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.801136 8491 leader_election.cc:288] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023), 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673)
I20260927 05:13:35.806438 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.801406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45481 (local address 127.0.97.2:42023)
0927 05:13:35.801708 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.801717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.801739 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:13:35.801971 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.801975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.802031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.802138 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:35.802520 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.803307 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.804822 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.805574 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.805715 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.805815 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.806074 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.806187 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 05:13:35.806435 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.801395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.97.3:45481)
0927 05:13:35.801658 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.801676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.801803 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.802155 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.802160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.802189 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:35.802392 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.802401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.803421 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.803429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.804678 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.804689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.805722 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.805727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.805915 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.805937 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.806036 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.806160 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":42}
I20260927 05:13:35.807497 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.801681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49741 (local address 127.0.97.1:37673)
0927 05:13:35.801843 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.801869 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.802532 (+ 663us) server_negotiation.cc:407] Connection header received
0927 05:13:35.802711 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.802714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.802782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.802868 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:35.803414 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.804060 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.805679 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.806548 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.806683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.806784 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.807151 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.807271 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:13:35.807593 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.801515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37673 (local address 127.0.97.3:49741)
0927 05:13:35.802436 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:35.802451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:35.802582 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:35.802893 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:35.802897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:35.802919 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:35.803283 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.803291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.804183 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.804188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.805533 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:35.805543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.806778 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.806784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:35.806953 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.806984 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 05:13:35.807127 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 05:13:35.807243 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":37,"thread_start_us":618,"threads_started":1}
I20260927 05:13:35.807432 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0f2d63ae9b464cad5fa3397a68541d" candidate_uuid: "81dd4404cf41406bb590fbe8895c95e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4a0567a984545990f322e7b5be382" is_pre_election: true
I20260927 05:13:35.808040 8272 raft_consensus.cc:2454] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81dd4404cf41406bb590fbe8895c95e5 in term 0.
I20260927 05:13:35.808579 8135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0f2d63ae9b464cad5fa3397a68541d" candidate_uuid: "81dd4404cf41406bb590fbe8895c95e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817" is_pre_election: true
I20260927 05:13:35.808669 8344 leader_election.cc:302] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 81dd4404cf41406bb590fbe8895c95e5; no voters:
I20260927 05:13:35.809000 8491 raft_consensus.cc:2781] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:35.809001 8135 raft_consensus.cc:2454] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81dd4404cf41406bb590fbe8895c95e5 in term 0.
I20260927 05:13:35.809124 8491 raft_consensus.cc:482] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:35.809235 8491 raft_consensus.cc:3037] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:35.811512 8491 raft_consensus.cc:504] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.812093 8491 leader_election.cc:288] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [CANDIDATE]: Term 1 election: Requested vote from peers 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023), 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673)
I20260927 05:13:35.812438 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0f2d63ae9b464cad5fa3397a68541d" candidate_uuid: "81dd4404cf41406bb590fbe8895c95e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4a0567a984545990f322e7b5be382"
I20260927 05:13:35.812522 8135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0f2d63ae9b464cad5fa3397a68541d" candidate_uuid: "81dd4404cf41406bb590fbe8895c95e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817"
I20260927 05:13:35.812613 8272 raft_consensus.cc:3037] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:35.812721 8135 raft_consensus.cc:3037] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:35.814543 8272 raft_consensus.cc:2454] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81dd4404cf41406bb590fbe8895c95e5 in term 1.
I20260927 05:13:35.814716 8135 raft_consensus.cc:2454] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81dd4404cf41406bb590fbe8895c95e5 in term 1.
I20260927 05:13:35.814916 8344 leader_election.cc:302] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 81dd4404cf41406bb590fbe8895c95e5; no voters:
I20260927 05:13:35.815191 8491 raft_consensus.cc:2781] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:35.815593 8491 raft_consensus.cc:686] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 1 LEADER]: Becoming Leader. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Running, Role: LEADER
I20260927 05:13:35.816001 8491 consensus_queue.cc:237] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } }
I20260927 05:13:35.820138 7975 catalog_manager.cc:5495] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "81dd4404cf41406bb590fbe8895c95e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:35.887794 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:35.882738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.97.3:34897)
0927 05:13:35.882944 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:35.882950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:35.882970 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:13:35.883155 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:35.883158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:35.883243 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:35.883392 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:35.883728 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.884411 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.885697 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:35.886110 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:35.886237 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:35.886661 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:35.887216 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:35.887362 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:35.887462 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 05:13:35.887537 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:13:35.895339 8135 raft_consensus.cc:1261] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Refusing update from remote peer 81dd4404cf41406bb590fbe8895c95e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:35.895490 8272 raft_consensus.cc:1261] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Refusing update from remote peer 81dd4404cf41406bb590fbe8895c95e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:35.896374 8491 consensus_queue.cc:1035] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:35.896669 8495 consensus_queue.cc:1035] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:35.908185 8500 mvcc.cc:205] Tried to move back new op lower bound from 7333830721098539008 to 7333830720785723392. Current Snapshot: MvccSnapshot[applied={T|T < 7333830721098539008}]
I20260927 05:13:35.921622 7976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:35.921875 7976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:35.925285 7976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Sending DeleteTablet for 3 replicas of tablet 0a0f2d63ae9b464cad5fa3397a68541d
I20260927 05:13:35.926267 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260927 05:13:35.926266 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35220
I20260927 05:13:35.926554 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0f2d63ae9b464cad5fa3397a68541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34086
I20260927 05:13:35.932627 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:35.934741 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:35.946329 8112 tablet_service.cc:1467] Processing CreateTablet for tablet 23e4516eeec54301a70eed062b458568 (DEFAULT_TABLE table=test-table [id=266a32e287b9460fa5e08a6da9c881be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.946624 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 23e4516eeec54301a70eed062b458568 (DEFAULT_TABLE table=test-table [id=266a32e287b9460fa5e08a6da9c881be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.946933 8112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e4516eeec54301a70eed062b458568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.947162 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e4516eeec54301a70eed062b458568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.951647 8251 tablet_service.cc:1467] Processing CreateTablet for tablet 23e4516eeec54301a70eed062b458568 (DEFAULT_TABLE table=test-table [id=266a32e287b9460fa5e08a6da9c881be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:35.952183 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e4516eeec54301a70eed062b458568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:35.954334 8489 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Bootstrap starting.
I20260927 05:13:35.954381 8486 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Bootstrap starting.
I20260927 05:13:35.956084 8489 tablet_bootstrap.cc:654] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.956110 8486 tablet_bootstrap.cc:654] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.958643 8485 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Bootstrap starting.
I20260927 05:13:35.960278 8485 tablet_bootstrap.cc:654] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:35.974386 8485 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: No bootstrap required, opened a new log
I20260927 05:13:35.974579 8485 ts_tablet_manager.cc:1397] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260927 05:13:35.975445 8485 raft_consensus.cc:348] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.975705 8485 raft_consensus.cc:374] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.975839 8485 raft_consensus.cc:729] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.976155 8485 consensus_queue.cc:260] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.976862 8485 ts_tablet_manager.cc:1428] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:35.979753 8508 tablet_replica.cc:330] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: stopping tablet replica
I20260927 05:13:35.980131 8510 tablet_replica.cc:330] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: stopping tablet replica
I20260927 05:13:35.980638 8508 raft_consensus.cc:2229] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:35.982476 8508 raft_consensus.cc:2258] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:35.981633 8486 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: No bootstrap required, opened a new log
I20260927 05:13:35.982786 8486 ts_tablet_manager.cc:1397] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.008s
I20260927 05:13:35.983582 8510 raft_consensus.cc:2229] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:35.983701 8486 raft_consensus.cc:348] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.983924 8486 raft_consensus.cc:374] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.984040 8486 raft_consensus.cc:729] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.984354 8510 raft_consensus.cc:2258] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:35.984331 8486 consensus_queue.cc:260] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.985196 8489 tablet_bootstrap.cc:492] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: No bootstrap required, opened a new log
I20260927 05:13:35.985354 8489 ts_tablet_manager.cc:1397] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.004s
I20260927 05:13:35.985816 8486 ts_tablet_manager.cc:1428] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:13:35.986153 8489 raft_consensus.cc:348] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.986373 8489 raft_consensus.cc:374] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:35.986492 8489 raft_consensus.cc:729] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Initialized, Role: FOLLOWER
I20260927 05:13:35.986820 8509 tablet_replica.cc:330] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: stopping tablet replica
I20260927 05:13:35.986716 8489 consensus_queue.cc:260] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:35.987241 8489 ts_tablet_manager.cc:1428] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:35.987378 8509 raft_consensus.cc:2229] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:35.987814 8509 raft_consensus.cc:2258] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:35.989818 8509 ts_tablet_manager.cc:1905] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:35.991669 8508 ts_tablet_manager.cc:1905] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:35.996795 8509 ts_tablet_manager.cc:1918] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:35.997028 8509 log.cc:1196] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/0a0f2d63ae9b464cad5fa3397a68541d
I20260927 05:13:35.997637 8509 ts_tablet_manager.cc:1939] T 0a0f2d63ae9b464cad5fa3397a68541d P 1be4a0567a984545990f322e7b5be382: Deleting consensus metadata
I20260927 05:13:35.998313 8510 ts_tablet_manager.cc:1905] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:35.999375 7962 catalog_manager.cc:4841] TS 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023): tablet 0a0f2d63ae9b464cad5fa3397a68541d (table test-table [id=0e5293643db742218c0292b236a3a5e4]) successfully deleted
I20260927 05:13:36.003788 8510 ts_tablet_manager.cc:1918] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:36.006397 8510 log.cc:1196] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/0a0f2d63ae9b464cad5fa3397a68541d
I20260927 05:13:36.006865 8510 ts_tablet_manager.cc:1939] T 0a0f2d63ae9b464cad5fa3397a68541d P 44729f2a086e4455b0db34f7d3404817: Deleting consensus metadata
I20260927 05:13:36.007284 8508 ts_tablet_manager.cc:1918] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:36.007504 8508 log.cc:1196] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/0a0f2d63ae9b464cad5fa3397a68541d
I20260927 05:13:36.008025 8508 ts_tablet_manager.cc:1939] T 0a0f2d63ae9b464cad5fa3397a68541d P 81dd4404cf41406bb590fbe8895c95e5: Deleting consensus metadata
I20260927 05:13:36.008581 7960 catalog_manager.cc:4841] TS 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673): tablet 0a0f2d63ae9b464cad5fa3397a68541d (table test-table [id=0e5293643db742218c0292b236a3a5e4]) successfully deleted
I20260927 05:13:36.009418 7961 catalog_manager.cc:4841] TS 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897): tablet 0a0f2d63ae9b464cad5fa3397a68541d (table test-table [id=0e5293643db742218c0292b236a3a5e4]) successfully deleted
I20260927 05:13:36.030673 8492 raft_consensus.cc:482] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:36.030866 8492 raft_consensus.cc:504] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.032755 8492 leader_election.cc:288] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.039870 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.033158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47533 (local address 127.0.97.1:37673)
0927 05:13:36.033324 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.033331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.034002 (+ 671us) server_negotiation.cc:407] Connection header received
0927 05:13:36.034292 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.034298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.034384 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.034475 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:36.035059 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.036171 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.037982 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.038857 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.038989 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.039106 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.039472 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.039601 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:13:36.039886 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.032966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37673 (local address 127.0.97.2:47533)
0927 05:13:36.033888 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:36.033908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:36.034124 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:36.034497 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:36.034502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:36.034529 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:36.034820 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.034831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.036305 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.036311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.037830 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.037841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.039084 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.039090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.039279 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.039313 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:13:36.039428 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 05:13:36.039557 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":614,"threads_started":1}
I20260927 05:13:36.040746 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e4516eeec54301a70eed062b458568" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817" is_pre_election: true
I20260927 05:13:36.041047 8134 raft_consensus.cc:2454] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.041133 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.033200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42107 (local address 127.0.97.3:34897)
0927 05:13:36.033459 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.033465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.034378 (+ 913us) server_negotiation.cc:407] Connection header received
0927 05:13:36.034783 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.034788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.034869 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.034930 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:36.035687 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.036547 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.039019 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.040192 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.040314 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.040407 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.040739 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.040864 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 05:13:36.041177 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.032966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34897 (local address 127.0.97.2:42107)
0927 05:13:36.034276 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:36.034294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:36.034402 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:36.035133 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:36.035137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:36.035159 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:36.035518 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.035531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.036683 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.036689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.038801 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.038815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.040386 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.040392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.040568 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.040601 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:13:36.040699 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:13:36.040818 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":303,"thread_start_us":678,"threads_started":1}
I20260927 05:13:36.041755 8203 leader_election.cc:302] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:36.042140 8492 raft_consensus.cc:2781] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:36.041923 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e4516eeec54301a70eed062b458568" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5" is_pre_election: true
I20260927 05:13:36.042260 8492 raft_consensus.cc:482] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:36.042347 8492 raft_consensus.cc:3037] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.042402 8416 raft_consensus.cc:2454] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.044600 8492 raft_consensus.cc:504] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.045122 8492 leader_election.cc:288] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Requested vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.045588 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e4516eeec54301a70eed062b458568" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5"
I20260927 05:13:36.045595 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e4516eeec54301a70eed062b458568" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817"
I20260927 05:13:36.045768 8416 raft_consensus.cc:3037] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.045780 8134 raft_consensus.cc:3037] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.047729 8134 raft_consensus.cc:2454] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.048092 8203 leader_election.cc:302] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:36.048344 8492 raft_consensus.cc:2781] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:36.048825 8492 raft_consensus.cc:686] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 1 LEADER]: Becoming Leader. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Running, Role: LEADER
I20260927 05:13:36.049027 8416 raft_consensus.cc:2454] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.049260 8492 consensus_queue.cc:237] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.062793 7976 catalog_manager.cc:5495] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be4a0567a984545990f322e7b5be382 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } health_report { overall_health: UNKNOWN } } }
Sep 27 05:13:36 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486013, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:36.110126 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.097347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.0.97.62:40995)
0927 05:13:36.098191 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.098195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.098207 (+ 12us) server_negotiation.cc:407] Connection header received
0927 05:13:36.098242 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.098245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.098294 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.098396 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:36.099571 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.100341 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.101584 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.101982 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.102097 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.105086 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:36.105090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:36.105093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:36.105118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:36.107665 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:36.108290 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:36.108298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:36.108303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:36.108407 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:36.108747 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:36.108752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:36.108755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:36.108994 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:36.109130 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.109479 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.109700 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":713,"thread_start_us":604,"threads_started":1}
I20260927 05:13:36.110208 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.097245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40995 (local address 127.0.0.1:55288)
0927 05:13:36.097862 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:36.097888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:36.098046 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:36.098420 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:36.098424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:36.099232 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:36.099447 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.099457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.100485 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.100490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.101457 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.101467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.102107 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.102112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.102293 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.102877 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:36.102893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:36.104904 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:36.107779 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:36.107791 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:36.107799 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:36.108188 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:36.108514 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:36.108521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:36.108524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:36.108647 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:36.109105 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:36.109112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:36.109264 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 05:13:36.109418 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 05:13:36.109812 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":135,"thread_start_us":417,"threads_started":1}
I20260927 05:13:36.114985 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55288:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:13:36.115991 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:36.122073 8113 tablet_service.cc:1467] Processing CreateTablet for tablet a52050cca93b497f9760735065a30bb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37594d1cd17a40aa9bac85c911533f82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:13:36.122172 8252 tablet_service.cc:1467] Processing CreateTablet for tablet a52050cca93b497f9760735065a30bb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37594d1cd17a40aa9bac85c911533f82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:13:36.122624 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52050cca93b497f9760735065a30bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.122658 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52050cca93b497f9760735065a30bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.128000 8485 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: Bootstrap starting.
I20260927 05:13:36.129185 8485 tablet_bootstrap.cc:654] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.129674 8489 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: Bootstrap starting.
I20260927 05:13:36.130718 8396 tablet_service.cc:1467] Processing CreateTablet for tablet a52050cca93b497f9760735065a30bb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37594d1cd17a40aa9bac85c911533f82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:13:36.131292 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52050cca93b497f9760735065a30bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.131712 8489 tablet_bootstrap.cc:654] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.139986 8486 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: Bootstrap starting.
I20260927 05:13:36.140439 8485 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: No bootstrap required, opened a new log
I20260927 05:13:36.140578 8485 ts_tablet_manager.cc:1397] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20260927 05:13:36.141319 8485 raft_consensus.cc:348] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.141536 8485 raft_consensus.cc:374] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.141613 8485 raft_consensus.cc:729] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.141711 8486 tablet_bootstrap.cc:654] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.141852 8485 consensus_queue.cc:260] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.142318 8492 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.142590 8492 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.142462 8485 ts_tablet_manager.cc:1428] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 05:13:36.147857 8486 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: No bootstrap required, opened a new log
I20260927 05:13:36.148048 8486 ts_tablet_manager.cc:1397] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260927 05:13:36.148672 8489 tablet_bootstrap.cc:492] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: No bootstrap required, opened a new log
I20260927 05:13:36.148834 8489 ts_tablet_manager.cc:1397] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.002s
I20260927 05:13:36.148801 8486 raft_consensus.cc:348] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.149032 8486 raft_consensus.cc:374] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.149116 8486 raft_consensus.cc:729] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.149348 8486 consensus_queue.cc:260] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.149546 8489 raft_consensus.cc:348] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.149765 8489 raft_consensus.cc:374] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.149853 8489 raft_consensus.cc:729] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.150029 8495 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.150343 8495 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.150182 8489 consensus_queue.cc:260] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.150758 8489 ts_tablet_manager.cc:1428] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 05:13:36.151090 8493 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.151430 8493 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.152779 8486 ts_tablet_manager.cc:1428] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 05:13:36.322295 8492 raft_consensus.cc:482] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:36.322526 8492 raft_consensus.cc:504] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.323318 8492 leader_election.cc:288] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.323940 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52050cca93b497f9760735065a30bb4" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817" is_pre_election: true
I20260927 05:13:36.324108 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52050cca93b497f9760735065a30bb4" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5" is_pre_election: true
I20260927 05:13:36.324204 8134 raft_consensus.cc:2454] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.324391 8416 raft_consensus.cc:2454] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.324733 8203 leader_election.cc:302] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:36.325052 8492 raft_consensus.cc:2781] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:36.325171 8492 raft_consensus.cc:482] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:36.325311 8492 raft_consensus.cc:3037] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.327378 8492 raft_consensus.cc:504] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.327966 8492 leader_election.cc:288] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Requested vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.328275 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52050cca93b497f9760735065a30bb4" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817"
I20260927 05:13:36.328421 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52050cca93b497f9760735065a30bb4" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5"
I20260927 05:13:36.328508 8134 raft_consensus.cc:3037] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.328655 8416 raft_consensus.cc:3037] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.331444 8134 raft_consensus.cc:2454] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.331687 8416 raft_consensus.cc:2454] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.332067 8204 leader_election.cc:302] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 81dd4404cf41406bb590fbe8895c95e5; no voters:
I20260927 05:13:36.332319 8492 raft_consensus.cc:2781] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:36.332487 8492 raft_consensus.cc:686] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [term 1 LEADER]: Becoming Leader. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Running, Role: LEADER
I20260927 05:13:36.332741 8492 consensus_queue.cc:237] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.334196 8516 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: TxnStatusTablet state changed. Reason: New leader 1be4a0567a984545990f322e7b5be382. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.334513 8516 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:13:36.335381 8529 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:13:36.335608 8529 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:13:36.335691 7974 catalog_manager.cc:5495] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be4a0567a984545990f322e7b5be382 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:36.384720 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.377791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.0.1:35224)
0927 05:13:36.378017 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:36.378038 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:36.378176 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:36.379246 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:36.379249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:36.379260 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:13:36.379410 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.379416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.380921 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.380924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.381908 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.381918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.382847 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.382851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.382993 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.383112 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:13:36.384332 (+ 1220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:13:36.384345 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 05:13:36.384395 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 05:13:36.384498 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 05:13:36.384891 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.377926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.97.2:42023)
0927 05:13:36.378786 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.378792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.378815 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:13:36.378908 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.378913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.379053 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.379217 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:36.379545 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.380809 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.382038 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.382710 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.382834 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.383487 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:36.384226 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:36.384376 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.384444 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.384519 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":513,"threads_started":1}
I20260927 05:13:36.388903 8134 raft_consensus.cc:1261] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Refusing update from remote peer 1be4a0567a984545990f322e7b5be382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:36.388950 8416 raft_consensus.cc:1261] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Refusing update from remote peer 1be4a0567a984545990f322e7b5be382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:36.389557 8516 consensus_queue.cc:1035] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:36.389886 8492 consensus_queue.cc:1035] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:36.397845 8495 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: TxnStatusTablet state changed. Reason: New leader 1be4a0567a984545990f322e7b5be382. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.398145 8495 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.400023 8492 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.400435 8492 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:13:36.403759 8493 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: TxnStatusTablet state changed. Reason: New leader 1be4a0567a984545990f322e7b5be382. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.403879 8499 mvcc.cc:205] Tried to move back new op lower bound from 7333830723121348608 to 7333830722900815872. Current Snapshot: MvccSnapshot[applied={T|T < 7333830723121348608 or (T in {7333830723121348608})}]
I20260927 05:13:36.404026 8493 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.404750 8495 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.404969 8495 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 81dd4404cf41406bb590fbe8895c95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.405869 8516 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.406225 8516 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 1be4a0567a984545990f322e7b5be382: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:13:36.411141 8493 tablet_replica.cc:439] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } } }
I20260927 05:13:36.411365 8493 tablet_replica.cc:442] T a52050cca93b497f9760735065a30bb4 P 44729f2a086e4455b0db34f7d3404817: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:13:36.424858 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.410531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.97.2:42023)
0927 05:13:36.410681 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.410687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.419553 (+ 8866us) server_negotiation.cc:407] Connection header received
0927 05:13:36.419714 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.419718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.419773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.419844 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:36.420315 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.421045 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.422928 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.423397 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.423548 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.424059 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:36.424158 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:36.424249 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.424451 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.424525 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:13:36.438926 8535 consensus_queue.cc:1035] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:36.440631 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.434636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60431 (local address 127.0.97.2:42023)
0927 05:13:36.434781 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.434785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.435421 (+ 636us) server_negotiation.cc:407] Connection header received
0927 05:13:36.435599 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.435608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.435673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.435752 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:36.436194 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.436884 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.438489 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.439175 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.439292 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.439837 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:36.439916 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:36.440019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.440169 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.440255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":37}
I20260927 05:13:36.440732 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.434728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.97.2:60431)
0927 05:13:36.435333 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:36.435347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:36.435495 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:36.435761 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:36.435765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:36.435774 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:13:36.436074 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.436081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.437009 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.437014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.438364 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:36.438377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.439439 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.439447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:36.439599 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.439670 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:13:36.440017 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:13:36.440035 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 05:13:36.440151 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:13:36.440292 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":2,"thread_start_us":358,"threads_started":1}
I20260927 05:13:36.449890 8416 raft_consensus.cc:1261] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Refusing update from remote peer 1be4a0567a984545990f322e7b5be382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:36.450326 8535 consensus_queue.cc:1035] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:36.462522 8498 mvcc.cc:205] Tried to move back new op lower bound from 7333830723374309376 to 7333830721741762560. Current Snapshot: MvccSnapshot[applied={T|T < 7333830723374309376}]
I20260927 05:13:36.541714 7974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:36.541972 7974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:36.544734 7974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Sending DeleteTablet for 3 replicas of tablet 23e4516eeec54301a70eed062b458568
I20260927 05:13:36.545442 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e4516eeec54301a70eed062b458568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34086
I20260927 05:13:36.545642 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e4516eeec54301a70eed062b458568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260927 05:13:36.545764 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e4516eeec54301a70eed062b458568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35220
I20260927 05:13:36.556756 8545 tablet_replica.cc:330] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: stopping tablet replica
I20260927 05:13:36.557158 8545 raft_consensus.cc:2229] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:36.557617 8545 raft_consensus.cc:2258] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:36.557909 8546 tablet_replica.cc:330] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: stopping tablet replica
I20260927 05:13:36.558223 8546 raft_consensus.cc:2229] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:36.558476 8546 raft_consensus.cc:2258] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:36.558774 8545 ts_tablet_manager.cc:1905] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:36.559304 8547 tablet_replica.cc:330] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: stopping tablet replica
I20260927 05:13:36.559422 8546 ts_tablet_manager.cc:1905] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:36.566591 8546 ts_tablet_manager.cc:1918] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:36.566720 8545 ts_tablet_manager.cc:1918] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:36.566771 8546 log.cc:1196] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/23e4516eeec54301a70eed062b458568
I20260927 05:13:36.567091 8547 raft_consensus.cc:2229] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:36.567247 8546 ts_tablet_manager.cc:1939] T 23e4516eeec54301a70eed062b458568 P 81dd4404cf41406bb590fbe8895c95e5: Deleting consensus metadata
I20260927 05:13:36.567683 8547 raft_consensus.cc:2258] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:36.568300 7961 catalog_manager.cc:4841] TS 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897): tablet 23e4516eeec54301a70eed062b458568 (table test-table [id=266a32e287b9460fa5e08a6da9c881be]) successfully deleted
I20260927 05:13:36.568794 8547 ts_tablet_manager.cc:1905] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:36.572518 8545 log.cc:1196] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/23e4516eeec54301a70eed062b458568
I20260927 05:13:36.573009 8545 ts_tablet_manager.cc:1939] T 23e4516eeec54301a70eed062b458568 P 44729f2a086e4455b0db34f7d3404817: Deleting consensus metadata
I20260927 05:13:36.574146 7960 catalog_manager.cc:4841] TS 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673): tablet 23e4516eeec54301a70eed062b458568 (table test-table [id=266a32e287b9460fa5e08a6da9c881be]) successfully deleted
I20260927 05:13:36.576728 8547 ts_tablet_manager.cc:1918] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:36.576864 8547 log.cc:1196] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/23e4516eeec54301a70eed062b458568
I20260927 05:13:36.577265 8547 ts_tablet_manager.cc:1939] T 23e4516eeec54301a70eed062b458568 P 1be4a0567a984545990f322e7b5be382: Deleting consensus metadata
I20260927 05:13:36.578090 7962 catalog_manager.cc:4841] TS 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023): tablet 23e4516eeec54301a70eed062b458568 (table test-table [id=266a32e287b9460fa5e08a6da9c881be]) successfully deleted
Sep 27 05:13:36 dist-test-slave-1t6p krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:13:36 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:36 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:36.615576 8524 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:36.600916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.0.97.62:40995)
0927 05:13:36.601117 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.601126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.601530 (+ 404us) server_negotiation.cc:407] Connection header received
0927 05:13:36.601721 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.601730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.601810 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.601911 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:36.603209 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.603915 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.605139 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.605559 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.605682 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.615289 (+ 9607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55298: BlockingRecv error: recv got EOF from 127.0.0.1:55298 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 05:13:36 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:13:36.632781 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:36.618181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.0.97.62:40995)
0927 05:13:36.618351 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:36.618359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:36.618646 (+ 287us) server_negotiation.cc:407] Connection header received
0927 05:13:36.618780 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:36.618785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:36.618865 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:36.618974 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:36.620076 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.621195 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.622404 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:36.623060 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:36.623189 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:36.626754 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:36.626761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:36.626766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:36.626813 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:36.630099 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:36.630811 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:36.630818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:36.630822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:36.630951 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:36.631452 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:36.631458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:36.631461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:36.631706 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:36.631977 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:36.632270 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 05:13:36.632477 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:13:36.636006 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:36.636572 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:36.643106 8113 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9a000668ff441ba54eb45bf634c002 (DEFAULT_TABLE table=test-table [id=c64e4d2a085044a4aa5f31f038e9695c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:36.643532 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9a000668ff441ba54eb45bf634c002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.643463 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9a000668ff441ba54eb45bf634c002 (DEFAULT_TABLE table=test-table [id=c64e4d2a085044a4aa5f31f038e9695c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:36.643592 8252 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9a000668ff441ba54eb45bf634c002 (DEFAULT_TABLE table=test-table [id=c64e4d2a085044a4aa5f31f038e9695c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:36.643859 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9a000668ff441ba54eb45bf634c002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.643977 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9a000668ff441ba54eb45bf634c002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:36.649999 8489 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Bootstrap starting.
I20260927 05:13:36.650326 8486 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Bootstrap starting.
I20260927 05:13:36.651667 8489 tablet_bootstrap.cc:654] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.651818 8486 tablet_bootstrap.cc:654] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.659646 8560 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Bootstrap starting.
I20260927 05:13:36.660678 8489 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: No bootstrap required, opened a new log
I20260927 05:13:36.660863 8489 ts_tablet_manager.cc:1397] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260927 05:13:36.661653 8489 raft_consensus.cc:348] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.661887 8560 tablet_bootstrap.cc:654] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:36.662043 8489 raft_consensus.cc:374] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.662204 8489 raft_consensus.cc:729] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.662509 8489 consensus_queue.cc:260] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.663285 8489 ts_tablet_manager.cc:1428] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 05:13:36.669767 8486 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: No bootstrap required, opened a new log
I20260927 05:13:36.669970 8486 ts_tablet_manager.cc:1397] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20260927 05:13:36.670202 8560 tablet_bootstrap.cc:492] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: No bootstrap required, opened a new log
I20260927 05:13:36.670349 8560 ts_tablet_manager.cc:1397] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260927 05:13:36.670740 8486 raft_consensus.cc:348] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.671030 8486 raft_consensus.cc:374] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.670985 8560 raft_consensus.cc:348] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.671175 8486 raft_consensus.cc:729] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.671271 8560 raft_consensus.cc:374] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:36.671407 8560 raft_consensus.cc:729] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Initialized, Role: FOLLOWER
I20260927 05:13:36.671396 8486 consensus_queue.cc:260] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.671654 8560 consensus_queue.cc:260] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.672155 8560 ts_tablet_manager.cc:1428] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:36.672982 8486 ts_tablet_manager.cc:1428] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:13:36.956385 8492 raft_consensus.cc:482] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:36.956591 8492 raft_consensus.cc:504] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.957218 8492 leader_election.cc:288] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.957717 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9a000668ff441ba54eb45bf634c002" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817" is_pre_election: true
I20260927 05:13:36.957851 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9a000668ff441ba54eb45bf634c002" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5" is_pre_election: true
I20260927 05:13:36.957994 8134 raft_consensus.cc:2454] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.958079 8416 raft_consensus.cc:2454] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 0.
I20260927 05:13:36.958494 8203 leader_election.cc:302] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817, 81dd4404cf41406bb590fbe8895c95e5; no voters:
I20260927 05:13:36.958798 8492 raft_consensus.cc:2781] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:36.958909 8492 raft_consensus.cc:482] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:36.959092 8492 raft_consensus.cc:3037] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.961179 8492 raft_consensus.cc:504] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.961755 8492 leader_election.cc:288] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Requested vote from peers 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:36.962096 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9a000668ff441ba54eb45bf634c002" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44729f2a086e4455b0db34f7d3404817"
I20260927 05:13:36.962174 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9a000668ff441ba54eb45bf634c002" candidate_uuid: "1be4a0567a984545990f322e7b5be382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5"
I20260927 05:13:36.962316 8134 raft_consensus.cc:3037] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.962344 8416 raft_consensus.cc:3037] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:36.965202 8134 raft_consensus.cc:2454] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.965215 8416 raft_consensus.cc:2454] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4a0567a984545990f322e7b5be382 in term 1.
I20260927 05:13:36.965638 8203 leader_election.cc:302] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:36.965941 8492 raft_consensus.cc:2781] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:36.966145 8492 raft_consensus.cc:686] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 1 LEADER]: Becoming Leader. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Running, Role: LEADER
I20260927 05:13:36.966445 8492 consensus_queue.cc:237] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:36.969167 7975 catalog_manager.cc:5495] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be4a0567a984545990f322e7b5be382 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1be4a0567a984545990f322e7b5be382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:37.023532 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.018560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.0.97.2:42023)
0927 05:13:37.019042 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.019049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.019066 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:13:37.019125 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.019128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.019198 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.019309 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:37.019660 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.020567 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.021856 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.022295 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.022443 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.022796 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.022916 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.023024 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.023173 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.023250 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":200,"threads_started":1}
I20260927 05:13:37.026832 8416 raft_consensus.cc:1261] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Refusing update from remote peer 1be4a0567a984545990f322e7b5be382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:37.027309 8134 raft_consensus.cc:1261] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Refusing update from remote peer 1be4a0567a984545990f322e7b5be382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:37.027694 8535 consensus_queue.cc:1035] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:37.027963 8492 consensus_queue.cc:1035] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:37.047597 7975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:37.047837 7975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:37.051910 7975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Sending DeleteTablet for 3 replicas of tablet 5f9a000668ff441ba54eb45bf634c002
I20260927 05:13:37.053200 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9a000668ff441ba54eb45bf634c002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260927 05:13:37.053467 8546 tablet_replica.cc:330] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: stopping tablet replica
I20260927 05:13:37.053668 8546 raft_consensus.cc:2229] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:37.053696 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9a000668ff441ba54eb45bf634c002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34086
I20260927 05:13:37.053882 8546 raft_consensus.cc:2258] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.053947 8545 tablet_replica.cc:330] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: stopping tablet replica
I20260927 05:13:37.054131 8545 raft_consensus.cc:2229] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:37.054324 8545 raft_consensus.cc:2258] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.054281 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9a000668ff441ba54eb45bf634c002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35220
I20260927 05:13:37.054549 8547 tablet_replica.cc:330] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: stopping tablet replica
I20260927 05:13:37.054762 8547 raft_consensus.cc:2229] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:37.055197 8545 ts_tablet_manager.cc:1905] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.055126 7975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:13:37.055716 7975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:13:37.055850 8547 raft_consensus.cc:2258] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.057571 8546 ts_tablet_manager.cc:1905] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.057600 8547 ts_tablet_manager.cc:1905] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.063861 8546 ts_tablet_manager.cc:1918] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:37.064116 8547 ts_tablet_manager.cc:1918] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:37.064266 8547 log.cc:1196] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/5f9a000668ff441ba54eb45bf634c002
I20260927 05:13:37.064724 8547 ts_tablet_manager.cc:1939] T 5f9a000668ff441ba54eb45bf634c002 P 1be4a0567a984545990f322e7b5be382: Deleting consensus metadata
I20260927 05:13:37.065711 8252 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (DEFAULT_TABLE table=test-table [id=18664ec53a7f44dbaedbb859d0119baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:37.066124 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:37.065778 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (DEFAULT_TABLE table=test-table [id=18664ec53a7f44dbaedbb859d0119baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:37.066660 8546 log.cc:1196] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/5f9a000668ff441ba54eb45bf634c002
I20260927 05:13:37.066891 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:37.067107 8546 ts_tablet_manager.cc:1939] T 5f9a000668ff441ba54eb45bf634c002 P 81dd4404cf41406bb590fbe8895c95e5: Deleting consensus metadata
I20260927 05:13:37.068600 7962 catalog_manager.cc:4841] TS 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023): tablet 5f9a000668ff441ba54eb45bf634c002 (table test-table [id=c64e4d2a085044a4aa5f31f038e9695c]) successfully deleted
I20260927 05:13:37.070827 8113 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (DEFAULT_TABLE table=test-table [id=18664ec53a7f44dbaedbb859d0119baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:13:37.071256 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:37.072331 8560 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Bootstrap starting.
I20260927 05:13:37.072973 8486 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Bootstrap starting.
I20260927 05:13:37.074005 8560 tablet_bootstrap.cc:654] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:37.074527 8486 tablet_bootstrap.cc:654] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:37.075496 7961 catalog_manager.cc:4841] TS 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897): tablet 5f9a000668ff441ba54eb45bf634c002 (table test-table [id=c64e4d2a085044a4aa5f31f038e9695c]) successfully deleted
I20260927 05:13:37.079016 8489 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Bootstrap starting.
I20260927 05:13:37.080654 8489 tablet_bootstrap.cc:654] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:37.082437 8545 ts_tablet_manager.cc:1918] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:13:37.082670 8545 log.cc:1196] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/5f9a000668ff441ba54eb45bf634c002
I20260927 05:13:37.083748 8545 ts_tablet_manager.cc:1939] T 5f9a000668ff441ba54eb45bf634c002 P 44729f2a086e4455b0db34f7d3404817: Deleting consensus metadata
I20260927 05:13:37.085624 7960 catalog_manager.cc:4841] TS 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673): tablet 5f9a000668ff441ba54eb45bf634c002 (table test-table [id=c64e4d2a085044a4aa5f31f038e9695c]) successfully deleted
I20260927 05:13:37.086264 8486 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: No bootstrap required, opened a new log
I20260927 05:13:37.086439 8486 ts_tablet_manager.cc:1397] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260927 05:13:37.086586 8560 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: No bootstrap required, opened a new log
I20260927 05:13:37.086791 8560 ts_tablet_manager.cc:1397] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20260927 05:13:37.087175 8486 raft_consensus.cc:348] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.087387 8486 raft_consensus.cc:374] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:37.087471 8486 raft_consensus.cc:729] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dd4404cf41406bb590fbe8895c95e5, State: Initialized, Role: FOLLOWER
I20260927 05:13:37.087564 8560 raft_consensus.cc:348] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.087811 8560 raft_consensus.cc:374] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:37.087692 8486 consensus_queue.cc:260] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.087896 8560 raft_consensus.cc:729] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4a0567a984545990f322e7b5be382, State: Initialized, Role: FOLLOWER
I20260927 05:13:37.088200 8486 ts_tablet_manager.cc:1428] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:37.088940 8489 tablet_bootstrap.cc:492] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: No bootstrap required, opened a new log
I20260927 05:13:37.089088 8489 ts_tablet_manager.cc:1397] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260927 05:13:37.089753 8489 raft_consensus.cc:348] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.089994 8489 raft_consensus.cc:374] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:37.089923 8560 consensus_queue.cc:260] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.090070 8489 raft_consensus.cc:729] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Initialized, Role: FOLLOWER
I20260927 05:13:37.090340 8489 consensus_queue.cc:260] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.090844 8489 ts_tablet_manager.cc:1428] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 05:13:37.092861 8560 ts_tablet_manager.cc:1428] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260927 05:13:37.251926 8493 raft_consensus.cc:482] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:13:37.252163 8493 raft_consensus.cc:504] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.253723 8493 leader_election.cc:288] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:37.259629 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.254067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51459 (local address 127.0.97.2:42023)
0927 05:13:37.254279 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.254285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.254620 (+ 335us) server_negotiation.cc:407] Connection header received
0927 05:13:37.254886 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.254890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.254949 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.255069 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:37.255597 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.256271 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.257943 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.258717 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.258852 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.258958 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.259255 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.259373 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:13:37.259867 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.254368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47341 (local address 127.0.97.3:34897)
0927 05:13:37.254884 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.254891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.254909 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:13:37.254979 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.254985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.255103 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.255199 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:13:37.255749 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.256405 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.258092 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.258861 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.259002 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.259141 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.259389 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.259526 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":280,"threads_started":1}
I20260927 05:13:37.259912 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.254070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34897 (local address 127.0.97.1:47341)
0927 05:13:37.254749 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:37.254767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:37.254877 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:37.255214 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:37.255218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:37.255243 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:37.255606 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.255614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.256533 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.256539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.257952 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.257964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.259084 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.259091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.259241 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.259270 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:13:37.259367 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:13:37.259489 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":3,"thread_start_us":415,"threads_started":1}
I20260927 05:13:37.259956 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.253887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.97.1:51459)
0927 05:13:37.254509 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:37.254532 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:37.254723 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:37.255090 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:37.255094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:37.255119 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:13:37.255428 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.255442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.256407 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.256412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.257791 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.257803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.258877 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.258884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.259108 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.259133 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 05:13:37.259230 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:13:37.259354 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":23,"thread_start_us":341,"threads_started":1}
I20260927 05:13:37.260506 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1e7fadcf414bc9b7fecbf45af8bdb1" candidate_uuid: "44729f2a086e4455b0db34f7d3404817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4a0567a984545990f322e7b5be382" is_pre_election: true
I20260927 05:13:37.260731 8272 raft_consensus.cc:2454] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44729f2a086e4455b0db34f7d3404817 in term 0.
I20260927 05:13:37.260712 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1e7fadcf414bc9b7fecbf45af8bdb1" candidate_uuid: "44729f2a086e4455b0db34f7d3404817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5" is_pre_election: true
I20260927 05:13:37.260921 8416 raft_consensus.cc:2454] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44729f2a086e4455b0db34f7d3404817 in term 0.
I20260927 05:13:37.261228 8065 leader_election.cc:302] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:37.261595 8493 raft_consensus.cc:2781] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:13:37.261708 8493 raft_consensus.cc:482] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:13:37.261802 8493 raft_consensus.cc:3037] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:37.263878 8493 raft_consensus.cc:504] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.264439 8493 leader_election.cc:288] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [CANDIDATE]: Term 1 election: Requested vote from peers 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023), 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897)
I20260927 05:13:37.264804 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1e7fadcf414bc9b7fecbf45af8bdb1" candidate_uuid: "44729f2a086e4455b0db34f7d3404817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4a0567a984545990f322e7b5be382"
I20260927 05:13:37.264873 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1e7fadcf414bc9b7fecbf45af8bdb1" candidate_uuid: "44729f2a086e4455b0db34f7d3404817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dd4404cf41406bb590fbe8895c95e5"
I20260927 05:13:37.264971 8272 raft_consensus.cc:3037] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:37.265077 8416 raft_consensus.cc:3037] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:37.267372 8272 raft_consensus.cc:2454] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44729f2a086e4455b0db34f7d3404817 in term 1.
I20260927 05:13:37.267635 8416 raft_consensus.cc:2454] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44729f2a086e4455b0db34f7d3404817 in term 1.
I20260927 05:13:37.267709 8065 leader_election.cc:302] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4a0567a984545990f322e7b5be382, 44729f2a086e4455b0db34f7d3404817; no voters:
I20260927 05:13:37.267995 8493 raft_consensus.cc:2781] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:37.268363 8493 raft_consensus.cc:686] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 1 LEADER]: Becoming Leader. State: Replica: 44729f2a086e4455b0db34f7d3404817, State: Running, Role: LEADER
I20260927 05:13:37.268702 8493 consensus_queue.cc:237] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } }
I20260927 05:13:37.271718 7974 catalog_manager.cc:5495] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44729f2a086e4455b0db34f7d3404817 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "44729f2a086e4455b0db34f7d3404817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44729f2a086e4455b0db34f7d3404817" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 } health_report { overall_health: UNKNOWN } } }
I20260927 05:13:37.293931 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.288432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.97.1:37673)
0927 05:13:37.288968 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.288972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.288990 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:13:37.289057 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.289060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.289131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.289261 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:37.289581 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.290340 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.291658 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.292082 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.292275 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.292690 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.293313 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.293458 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.293553 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.293623 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
I20260927 05:13:37.307282 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.302685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33829 (local address 127.0.97.2:42023)
0927 05:13:37.302816 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.302820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.303332 (+ 512us) server_negotiation.cc:407] Connection header received
0927 05:13:37.303475 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.303478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.303534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.303598 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:37.303956 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.304592 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.305792 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.306194 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.306325 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.306648 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.306781 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.306855 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.306980 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.307091 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:13:37.307356 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.302593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42023 (local address 127.0.97.1:33829)
0927 05:13:37.303254 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:37.303268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:37.303369 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:37.303620 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:37.303623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:37.303630 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:13:37.303851 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.303857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.304691 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.304694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.305668 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.305677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.306347 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.306352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.306466 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.306509 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.306867 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.306874 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:13:37.306928 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 05:13:37.307092 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":350,"threads_started":1}
I20260927 05:13:37.314500 8272 raft_consensus.cc:1261] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Refusing update from remote peer 44729f2a086e4455b0db34f7d3404817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:37.314500 8416 raft_consensus.cc:1261] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Refusing update from remote peer 44729f2a086e4455b0db34f7d3404817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:13:37.315126 8581 consensus_queue.cc:1035] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dd4404cf41406bb590fbe8895c95e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:37.315423 8493 consensus_queue.cc:1035] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be4a0567a984545990f322e7b5be382" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:13:37.338521 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37673 (local address 127.0.97.2:32851)
0927 05:13:37.333800 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:37.333815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:37.333946 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:37.334230 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:37.334234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:37.334245 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:13:37.334571 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.334586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.335680 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.335686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.336834 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:37.336845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.337536 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.337545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:37.337680 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.337711 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.338050 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.338060 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:13:37.338122 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 05:13:37.338212 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260927 05:13:37.338709 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.333298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32851 (local address 127.0.97.1:37673)
0927 05:13:37.333407 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.333413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.333903 (+ 490us) server_negotiation.cc:407] Connection header received
0927 05:13:37.334061 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.334066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.334133 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.334217 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:13:37.334696 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.335553 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.336964 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.337374 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.337500 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.337859 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:13:37.337954 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:13:37.338049 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.338142 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.338453 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 05:13:37.367865 7975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:37.368136 7975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:13:37.370738 7975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1975cfa807e244779bf40c2beee6b26f: Sending DeleteTablet for 3 replicas of tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1
I20260927 05:13:37.371390 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34086
I20260927 05:13:37.371601 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260927 05:13:37.371690 8545 tablet_replica.cc:330] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: stopping tablet replica
I20260927 05:13:37.371678 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:13:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35220
I20260927 05:13:37.371867 8546 tablet_replica.cc:330] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: stopping tablet replica
I20260927 05:13:37.371865 8545 raft_consensus.cc:2229] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:13:37.371928 8547 tablet_replica.cc:330] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: stopping tablet replica
I20260927 05:13:37.372005 8546 raft_consensus.cc:2229] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:37.372076 8547 raft_consensus.cc:2229] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:13:37.372184 8546 raft_consensus.cc:2258] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.372251 8547 raft_consensus.cc:2258] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.372334 8545 raft_consensus.cc:2258] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:13:37.373286 8546 ts_tablet_manager.cc:1905] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.373267 8547 ts_tablet_manager.cc:1905] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.373471 8545 ts_tablet_manager.cc:1905] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:13:37.379914 8546 ts_tablet_manager.cc:1918] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:37.380059 8546 log.cc:1196] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/5c1e7fadcf414bc9b7fecbf45af8bdb1
I20260927 05:13:37.380050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8047
I20260927 05:13:37.380286 8547 ts_tablet_manager.cc:1918] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:37.380496 8546 ts_tablet_manager.cc:1939] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 81dd4404cf41406bb590fbe8895c95e5: Deleting consensus metadata
I20260927 05:13:37.380517 8547 log.cc:1196] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/5c1e7fadcf414bc9b7fecbf45af8bdb1
I20260927 05:13:37.381554 7961 catalog_manager.cc:4841] TS 81dd4404cf41406bb590fbe8895c95e5 (127.0.97.3:34897): tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (table test-table [id=18664ec53a7f44dbaedbb859d0119baf]) successfully deleted
I20260927 05:13:37.381016 8547 ts_tablet_manager.cc:1939] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 1be4a0567a984545990f322e7b5be382: Deleting consensus metadata
I20260927 05:13:37.383858 8545 ts_tablet_manager.cc:1918] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:13:37.383914 7962 catalog_manager.cc:4841] TS 1be4a0567a984545990f322e7b5be382 (127.0.97.2:42023): tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (table test-table [id=18664ec53a7f44dbaedbb859d0119baf]) successfully deleted
I20260927 05:13:37.384035 8545 log.cc:1196] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/5c1e7fadcf414bc9b7fecbf45af8bdb1
I20260927 05:13:37.384549 8545 ts_tablet_manager.cc:1939] T 5c1e7fadcf414bc9b7fecbf45af8bdb1 P 44729f2a086e4455b0db34f7d3404817: Deleting consensus metadata
I20260927 05:13:37.385457 7960 catalog_manager.cc:4841] TS 44729f2a086e4455b0db34f7d3404817 (127.0.97.1:37673): tablet 5c1e7fadcf414bc9b7fecbf45af8bdb1 (table test-table [id=18664ec53a7f44dbaedbb859d0119baf]) successfully deleted
Sep 27 05:13:37 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 05:13:37.400846 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.382401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.0.97.1:37673)
0927 05:13:37.382585 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.382592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.384872 (+ 2280us) server_negotiation.cc:407] Connection header received
0927 05:13:37.385044 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.385049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.385141 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.385245 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:37.387044 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.387808 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.389524 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.389911 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.390004 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.394851 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:37.394861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:37.394875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:37.394903 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:37.398111 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.398869 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.398876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.398878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.399076 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.399441 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.399445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.399447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.399764 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:37.400051 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.400353 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.400509 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:13:37.401618 8178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:37.508203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8047
I20260927 05:13:37.527315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8187
Sep 27 05:13:37 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 05:13:37.541574 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.528066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.0.97.2:42023)
0927 05:13:37.528229 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.528236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.528278 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:13:37.528500 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.528505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.528572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.528667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:37.529786 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.530516 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.531815 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.532223 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.532348 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.536242 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:37.536253 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:37.536267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:37.536303 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:37.538814 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.539641 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.539649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.539653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.539769 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.540129 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.540134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.540137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.540498 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:37.540778 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.541059 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.541272 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:13:37.542173 8317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:37.687947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8187
I20260927 05:13:37.712589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8326
Sep 27 05:13:37 dist-test-slave-1t6p krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 05:13:37.727169 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:37.713364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.97.3:34897)
0927 05:13:37.713552 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:37.713557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:37.713594 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:13:37.713785 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:37.713789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:37.713850 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:37.713966 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:37.715394 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.716185 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.717350 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:37.717748 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:37.717873 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:37.721867 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:37.721885 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:37.721905 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:37.721935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:37.724761 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.725327 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.725334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.725336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.725463 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:37.725854 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:37.725857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:37.725859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:37.726184 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:37.726410 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:37.726604 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 05:13:37.726779 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:13:37.727761 8456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:37.844672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8326
I20260927 05:13:37.865625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 7944
I20260927 05:13:37.866851 8016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:37.978363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 7944
2026-09-27T05:13:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5789 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:38 dist-test-slave-1t6p krb5kdc[8602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:38 dist-test-slave-1t6p krb5kdc[8602](info): set up 2 sockets
Sep 27 05:13:38 dist-test-slave-1t6p krb5kdc[8602](info): commencing operation
krb5kdc: starting...
W20260927 05:13:39.059988 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:39 dist-test-slave-1t6p krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:39.256311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43583
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:39.561602 8615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:39.561939 8615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:39.562091 8615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:39.568871 8615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:39.569000 8615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:39.569096 8615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:39.569190 8615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:39.569270 8615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:39.578397 8615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:39.579552 8615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:39.580763 8615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:39.592041 8623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:39.592077 8620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:39.592108 8621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:39.593608 8615 server_base.cc:1034] running on GCE node
I20260927 05:13:39.594316 8615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:39.595656 8615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:39.596890 8615 hybrid_clock.cc:648] HybridClock initialized: now 1790486019596830 us; error 47 us; skew 500 ppm
I20260927 05:13:39.597209 8615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T05:13:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1694 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:13:39 dist-test-slave-1t6p krb5kdc[8629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:13:39 dist-test-slave-1t6p krb5kdc[8629](info): set up 2 sockets
Sep 27 05:13:39 dist-test-slave-1t6p krb5kdc[8629](info): commencing operation
krb5kdc: starting...
W20260927 05:13:40.755880 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:13:40 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:13:40.951579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44777
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:41.254628 8642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:41.254885 8642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:41.254978 8642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:41.261873 8642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:41.261973 8642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:41.262018 8642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:41.262075 8642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:41.262117 8642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:41.271411 8642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:41.272524 8642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:41.273680 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:41.285104 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:41.285274 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:41.285583 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:41.285869 8642 server_base.cc:1034] running on GCE node
I20260927 05:13:41.286629 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:41.287986 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:41.289220 8642 hybrid_clock.cc:648] HybridClock initialized: now 1790486021289181 us; error 43 us; skew 500 ppm
I20260927 05:13:41.289541 8642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:41 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:41.292523 8642 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:41.293793 8642 webserver.cc:460] Webserver started at http://127.0.97.62:41941/ using document root <none> and password file <none>
I20260927 05:13:41.294365 8642 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:41.294482 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:41.294827 8642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:41.297621 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "630105165248458abfb6ee39b4bb6afe"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "3d6fdfee9f28e955f1e5705684297295"
server_key_iv: "3c3f3359a649c08327f90e060f0ab9ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.298368 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "630105165248458abfb6ee39b4bb6afe"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "3d6fdfee9f28e955f1e5705684297295"
server_key_iv: "3c3f3359a649c08327f90e060f0ab9ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.302927 8642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 05:13:41.306079 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:41.307261 8642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 05:13:41.307402 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "630105165248458abfb6ee39b4bb6afe"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "3d6fdfee9f28e955f1e5705684297295"
server_key_iv: "3c3f3359a649c08327f90e060f0ab9ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.307588 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:41.340931 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:41.344472 8642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:41.344759 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:41.361936 8642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44777
I20260927 05:13:41.362007 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44777 every 8 connection(s)
I20260927 05:13:41.363190 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:41.366648 8709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:13:41.368690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8642
I20260927 05:13:41.368907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:13:41.369491 388 external_mini_cluster.cc:1428] Setting key 1745f5c4b502c37fdbcf5a7cae0358bf
I20260927 05:13:41.377398 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Bootstrap starting.
I20260927 05:13:41.380859 8709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Neither blocks nor log segments found. Creating new log.
I20260927 05:13:41.382087 8709 log.cc:824] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:13:41 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:41.384864 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: No bootstrap required, opened a new log
I20260927 05:13:41.391227 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:41.371363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.0.97.62:44777)
0927 05:13:41.372087 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:41.372098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:41.372135 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:13:41.373668 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:41.373686 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:41.374027 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:41.374333 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:41.376003 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.377227 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.378940 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.379585 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.379708 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:41.384282 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:41.384290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:41.384303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:41.384341 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:41.387432 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:41.388094 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:41.388100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:41.388105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:41.388267 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:41.388645 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:41.388648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:41.388650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:41.389146 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:41.389448 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:41.389876 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:13:41.390238 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":344,"threads_started":1}
I20260927 05:13:41.392536 8709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:41.392875 8709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:13:41.392994 8709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630105165248458abfb6ee39b4bb6afe, State: Initialized, Role: FOLLOWER
I20260927 05:13:41.393513 8709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:41.393751 8709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:41.393893 8709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:41.394063 8709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:13:41.396087 8709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:41.396529 8709 leader_election.cc:302] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 630105165248458abfb6ee39b4bb6afe; no voters:
I20260927 05:13:41.397017 8709 leader_election.cc:288] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:13:41.397218 8714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:13:41.397850 8714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 LEADER]: Becoming Leader. State: Replica: 630105165248458abfb6ee39b4bb6afe, State: Running, Role: LEADER
I20260927 05:13:41.398418 8714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:41.399024 8709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:41.401232 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "630105165248458abfb6ee39b4bb6afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } } }
I20260927 05:13:41.401209 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630105165248458abfb6ee39b4bb6afe. Latest consensus state: current_term: 1 leader_uuid: "630105165248458abfb6ee39b4bb6afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } } }
I20260927 05:13:41.401645 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:41.401645 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:41.403786 8721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:41.409739 8721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:41.419688 8721 catalog_manager.cc:1347] Generated new cluster ID: 23311b9a28e24aef83c1c3eb8b8661c7
I20260927 05:13:41.419809 8721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:41.430660 8721 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:13:41.431854 8721 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:41.444654 8721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Generated new TSK 0
I20260927 05:13:41.445367 8721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:13:41.496526 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:41.812208 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:41.812466 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:41.812556 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:41.819934 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:41.820029 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:41.820145 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:41.829272 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:41.830335 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:41.831568 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:41.844076 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:41.844161 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:41.844130 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:41.845234 8735 server_base.cc:1034] running on GCE node
I20260927 05:13:41.845785 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:41.846652 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:41.847875 8735 hybrid_clock.cc:648] HybridClock initialized: now 1790486021847828 us; error 60 us; skew 500 ppm
I20260927 05:13:41.848160 8735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:41 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:41.857357 8735 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:13:41.858999 8735 webserver.cc:460] Webserver started at http://127.0.97.1:34557/ using document root <none> and password file <none>
I20260927 05:13:41.859699 8735 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:41.859834 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:41.860136 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:41.862799 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "18d7d164e7884e3891d7b73032ae84f0"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "ccf63ad027f3e33244878c2e94081b2b"
server_key_iv: "f59d090d2dd11cf88b88930a08ca1eeb"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.863601 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "18d7d164e7884e3891d7b73032ae84f0"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "ccf63ad027f3e33244878c2e94081b2b"
server_key_iv: "f59d090d2dd11cf88b88930a08ca1eeb"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.868158 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:13:41.871317 8750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:41.872478 8735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:13:41.872650 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "18d7d164e7884e3891d7b73032ae84f0"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "ccf63ad027f3e33244878c2e94081b2b"
server_key_iv: "f59d090d2dd11cf88b88930a08ca1eeb"
server_key_version: "encryptionkey@0"
I20260927 05:13:41.872843 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:41.887898 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:41.892022 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:41.892299 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:41.893181 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:41.894371 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:41.894470 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:41.894619 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:41.894692 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:41.926609 8735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39547
I20260927 05:13:41.926648 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39547 every 8 connection(s)
I20260927 05:13:41.927897 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:41.937372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8735
I20260927 05:13:41.937582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:13:41.938313 388 external_mini_cluster.cc:1428] Setting key e6dc10fa0dd9c9186eada604be223101
Sep 27 05:13:41 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:41.947796 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:41.930296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57153 (local address 127.0.97.62:44777)
0927 05:13:41.930693 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:41.930697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:41.931744 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 05:13:41.933452 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:41.933456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:41.933518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:41.933617 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:41.934940 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.935715 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.937356 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.937854 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.937949 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:41.941501 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:41.941512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:41.941520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:41.941574 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:41.944960 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:41.945612 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:41.945620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:41.945626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:41.945768 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:41.946167 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:41.946175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:41.946181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:41.946593 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:41.946741 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:41.947408 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 05:13:41.947553 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":210,"threads_started":1}
I20260927 05:13:41.949272 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:41.930867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:57153)
0927 05:13:41.931594 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:41.931629 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:41.933146 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:41.933803 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:41.933811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:41.934273 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:41.934806 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:41.934821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.935840 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.935844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:41.937143 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:41.937159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:41.938069 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:41.938078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:41.938380 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:41.939227 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:41.939247 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:41.941292 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:41.945088 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:41.945095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:41.945106 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:41.945490 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:41.945882 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:41.945885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:41.945887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:41.946065 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:41.946710 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:41.946716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:41.947059 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 05:13:41.947347 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 05:13:41.947879 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":316,"threads_started":1}
I20260927 05:13:41.950750 8863 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44777
I20260927 05:13:41.951071 8863 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:41.951725 8863 heartbeater.cc:511] Master 127.0.97.62:44777 requested a full tablet report, sending...
I20260927 05:13:41.953559 8674 ts_manager.cc:194] Registered new tserver with Master: 18d7d164e7884e3891d7b73032ae84f0 (127.0.97.1:39547)
I20260927 05:13:41.954917 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57153
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:13:41.984396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:42.293955 8869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:42.294260 8869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:42.294441 8869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:42.301837 8869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:42.301930 8869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:42.302035 8869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:42.311378 8869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:42.312668 8869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:42.313849 8869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:42.326160 8875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:42.326305 8874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:42.326283 8877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:42.327268 8869 server_base.cc:1034] running on GCE node
I20260927 05:13:42.327862 8869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:42.328734 8869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:42.329952 8869 hybrid_clock.cc:648] HybridClock initialized: now 1790486022329904 us; error 52 us; skew 500 ppm
I20260927 05:13:42.330255 8869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:42 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:42.333148 8869 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:13:42.334412 8869 webserver.cc:460] Webserver started at http://127.0.97.2:43719/ using document root <none> and password file <none>
I20260927 05:13:42.335085 8869 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:42.335218 8869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:42.335505 8869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:42.338145 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "638b67779a6f4523ae586481f674482a"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "2cc5c724b8d1edef96b4efb0c1d5f25e"
server_key_iv: "f6187661aaa618f997dfde7495d278ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.338855 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "638b67779a6f4523ae586481f674482a"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "2cc5c724b8d1edef96b4efb0c1d5f25e"
server_key_iv: "f6187661aaa618f997dfde7495d278ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.343519 8869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 05:13:42.346613 8884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.347991 8869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:42.348134 8869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "638b67779a6f4523ae586481f674482a"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "2cc5c724b8d1edef96b4efb0c1d5f25e"
server_key_iv: "f6187661aaa618f997dfde7495d278ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.348410 8869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:42.372556 8869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:42.376682 8869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:42.376955 8869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:42.377918 8869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:42.379207 8869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:42.379302 8869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.379432 8869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:42.379503 8869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.411407 8869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34083
I20260927 05:13:42.411482 8996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34083 every 8 connection(s)
I20260927 05:13:42.412727 8869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:42.421605 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 8869
I20260927 05:13:42.421780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:13:42.422194 388 external_mini_cluster.cc:1428] Setting key 06efed0e92fbc7c5bc9ec59aebffd874
Sep 27 05:13:42 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:42.432186 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:42.415069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54535 (local address 127.0.97.62:44777)
0927 05:13:42.415234 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:42.415238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:42.416298 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 05:13:42.418060 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:42.418066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:42.418144 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:42.418262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:42.419691 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.420429 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.421998 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.422455 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.422591 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:42.426400 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:42.426405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:42.426408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:42.426437 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:42.428921 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:42.429788 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:42.429792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:42.429794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:42.429907 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:42.430362 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:42.430365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:42.430367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:42.430610 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:42.430712 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:42.431540 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 05:13:42.431781 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:13:42.432757 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:42.415520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:54535)
0927 05:13:42.416156 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:42.416196 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:42.417788 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:42.418472 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:42.418479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:42.418938 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:42.419534 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:42.419550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.420593 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.420597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:42.421827 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:42.421864 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.422674 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.422682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:42.422895 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:42.423548 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:42.423575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:42.426190 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:42.429054 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:42.429065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:42.429081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:42.429678 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:42.430009 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:42.430016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:42.430022 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:42.430244 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:42.430741 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:42.430752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:42.431172 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 05:13:42.431396 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 05:13:42.431735 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
I20260927 05:13:42.434095 8997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44777
I20260927 05:13:42.434389 8997 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:42.435194 8997 heartbeater.cc:511] Master 127.0.97.62:44777 requested a full tablet report, sending...
I20260927 05:13:42.436683 8674 ts_manager.cc:194] Registered new tserver with Master: 638b67779a6f4523ae586481f674482a (127.0.97.2:34083)
I20260927 05:13:42.437457 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54535
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:13:42.469461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:42.777120 9002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:42.777462 9002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:42.777632 9002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:42.784653 9002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:42.784775 9002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:42.784940 9002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:42.794014 9002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:42.795307 9002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:42.796530 9002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:42.809707 9010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:42.809743 9008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:42.810052 9007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:42.810598 9002 server_base.cc:1034] running on GCE node
I20260927 05:13:42.811230 9002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:42.812089 9002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:42.813318 9002 hybrid_clock.cc:648] HybridClock initialized: now 1790486022813272 us; error 31 us; skew 500 ppm
I20260927 05:13:42.813643 9002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:42 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:42.816730 9002 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:13:42.818055 9002 webserver.cc:460] Webserver started at http://127.0.97.3:34423/ using document root <none> and password file <none>
I20260927 05:13:42.818701 9002 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:42.818827 9002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:42.819187 9002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:13:42.821853 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "ddf75da8269f4d8d8b2eb4e3620e1722"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "4f4d736b35036d6d99cbec3fce442da1"
server_key_iv: "87585d2ee267d51c599b167e96cd00dd"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.822599 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ddf75da8269f4d8d8b2eb4e3620e1722"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "4f4d736b35036d6d99cbec3fce442da1"
server_key_iv: "87585d2ee267d51c599b167e96cd00dd"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.827322 9002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 05:13:42.830476 9017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.831753 9002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:42.831924 9002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "ddf75da8269f4d8d8b2eb4e3620e1722"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "4f4d736b35036d6d99cbec3fce442da1"
server_key_iv: "87585d2ee267d51c599b167e96cd00dd"
server_key_version: "encryptionkey@0"
I20260927 05:13:42.832116 9002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:42.848109 9002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:42.851861 9002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:42.852126 9002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:42.852980 9002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:42.854214 9002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:42.854315 9002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.854458 9002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:42.854537 9002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:42.887048 9002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37677
I20260927 05:13:42.887125 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37677 every 8 connection(s)
I20260927 05:13:42.888312 9002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:42.896421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9002
I20260927 05:13:42.896631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:13:42.897029 388 external_mini_cluster.cc:1428] Setting key 656759411f294747b3e1c615e46e078b
Sep 27 05:13:42 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:13:42.906870 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:42.890679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37121 (local address 127.0.97.62:44777)
0927 05:13:42.890852 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:42.890856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:42.892035 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 05:13:42.893756 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:42.893761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:42.893845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:42.893952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:42.895433 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.896226 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.898034 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.898549 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.898688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:42.902028 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:42.902033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:42.902037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:42.902067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:42.904527 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:42.905144 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:42.905147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:42.905150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:42.905256 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:42.905589 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:42.905592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:42.905593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:42.905824 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:42.905940 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:42.906511 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 05:13:42.906651 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:13:42.907867 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:42.891196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:37121)
0927 05:13:42.891861 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:42.891906 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:42.893490 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:42.894175 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:42.894183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:42.894676 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:42.895274 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:42.895291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.896375 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.896385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:42.897824 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:42.897837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:42.898767 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:42.898774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:42.899092 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:42.899752 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:42.899783 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:42.901839 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:13:42.904641 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:42.904646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:42.904657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:42.905045 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:42.905340 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:13:42.905342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:13:42.905345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:13:42.905497 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:13:42.905935 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:13:42.905940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:13:42.906217 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 05:13:42.906426 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 05:13:42.906739 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":252,"threads_started":1}
I20260927 05:13:42.909124 9130 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44777
I20260927 05:13:42.909395 9130 heartbeater.cc:464] Registering TS with master...
I20260927 05:13:42.910108 9130 heartbeater.cc:511] Master 127.0.97.62:44777 requested a full tablet report, sending...
I20260927 05:13:42.911664 8674 ts_manager.cc:194] Registered new tserver with Master: ddf75da8269f4d8d8b2eb4e3620e1722 (127.0.97.3:37677)
I20260927 05:13:42.912214 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:13:42.912621 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37121
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 05:13:42.957664 8863 heartbeater.cc:503] Master 127.0.97.62:44777 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:13:43.039654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8735
Sep 27 05:13:43 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:13:43.054648 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:43.040832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.97.1:39547)
0927 05:13:43.041451 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:43.041458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:43.041484 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:13:43.041543 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:43.041551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:43.041758 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:43.041892 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:43.043126 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.044115 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.045296 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.045877 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.045970 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:43.049699 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:43.049710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:43.049723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:43.049752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:43.052465 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.053066 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.053073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.053075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.053191 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.053481 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.053484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.053486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.053783 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:43.053986 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:43.054203 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:13:43.054376 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":323,"threads_started":1}
I20260927 05:13:43.055883 8858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:43.141201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8735
I20260927 05:13:43.157810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8869
Sep 27 05:13:43 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:13:43.173491 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:43.158589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.0.97.2:34083)
0927 05:13:43.159134 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:43.159141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:43.159165 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:13:43.159222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:43.159228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:43.159432 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:43.159595 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:43.160800 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.161796 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.163062 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.163589 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.163735 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:43.167653 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:43.167664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:43.167676 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:43.167705 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:43.170316 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.171489 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.171497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.171501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.171624 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.172044 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.172048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.172050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.172399 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:43.172644 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:43.172898 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 05:13:43.173086 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260927 05:13:43.174671 8992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:43.256824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8869
I20260927 05:13:43.273886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9002
Sep 27 05:13:43 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:13:43.289566 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:43.274837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.0.97.3:37677)
0927 05:13:43.275503 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:43.275510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:43.275543 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:13:43.275607 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:43.275614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:43.275820 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:43.275999 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:43.277299 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.278376 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.279854 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:43.280410 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:43.280549 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:43.284686 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:43.284696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:43.284709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:43.284739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:43.287349 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.287910 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.287917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.287919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.288050 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:43.288337 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:43.288340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:43.288342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:43.288646 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:43.288866 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:43.289103 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 05:13:43.289283 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":369,"threads_started":1}
I20260927 05:13:43.290835 9125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:43.377185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9002
I20260927 05:13:43.392441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 8642
I20260927 05:13:43.393563 8704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:13:43.492342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 8642
I20260927 05:13:43.513455 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44777
--webserver_interface=127.0.97.62
--webserver_port=41941
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:13:43.824707 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:43.825016 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:43.825172 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:43.832402 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:13:43.832532 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:43.832651 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:43.832742 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:13:43.832834 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:13:43.842221 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:43.843465 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:43.844676 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:43.855852 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:43.855885 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:43.855990 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:43.856554 9147 server_base.cc:1034] running on GCE node
I20260927 05:13:43.857213 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:43.858565 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:43.859783 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790486023859739 us; error 58 us; skew 500 ppm
I20260927 05:13:43.860119 9147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:43 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:43.863112 9147 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:13:43.864451 9147 webserver.cc:460] Webserver started at http://127.0.97.62:41941/ using document root <none> and password file <none>
I20260927 05:13:43.865092 9147 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:43.865243 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:43.870255 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:13:43.872920 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:43.874114 9147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:13:43.874286 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "630105165248458abfb6ee39b4bb6afe"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "3d6fdfee9f28e955f1e5705684297295"
server_key_iv: "3c3f3359a649c08327f90e060f0ab9ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:43.875224 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:43.889535 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:43.893317 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:43.893571 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:43.911548 9147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44777
I20260927 05:13:43.911583 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44777 every 8 connection(s)
I20260927 05:13:43.912953 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:13:43.919582 9214 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:13:43.922621 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Bootstrap starting.
I20260927 05:13:43.922676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9147
I20260927 05:13:43.923522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39547
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44777
--webserver_port=34557
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:13:43.946607 9214 log.cc:824] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Log is configured to *not* fsync() on all Append() calls
I20260927 05:13:43.957059 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:13:43.957615 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Bootstrap complete.
I20260927 05:13:43.970608 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:43.971724 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630105165248458abfb6ee39b4bb6afe, State: Initialized, Role: FOLLOWER
I20260927 05:13:43.972393 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:43.972641 9214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:13:43.972766 9214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:13:43.972940 9214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:13:43.976600 9214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:43.977072 9214 leader_election.cc:302] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 630105165248458abfb6ee39b4bb6afe; no voters:
I20260927 05:13:43.977648 9214 leader_election.cc:288] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:13:43.977906 9218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:13:43.978948 9218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [term 2 LEADER]: Becoming Leader. State: Replica: 630105165248458abfb6ee39b4bb6afe, State: Running, Role: LEADER
I20260927 05:13:43.979568 9218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } }
I20260927 05:13:43.979791 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:13:43.981909 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "630105165248458abfb6ee39b4bb6afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } } }
I20260927 05:13:43.982236 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:43.984478 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630105165248458abfb6ee39b4bb6afe. Latest consensus state: current_term: 2 leader_uuid: "630105165248458abfb6ee39b4bb6afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630105165248458abfb6ee39b4bb6afe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44777 } } }
I20260927 05:13:43.984862 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe [sys.catalog]: This master's current role is: LEADER
I20260927 05:13:43.986001 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:13:43.993046 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:13:43.994374 9223 catalog_manager.cc:1259] Loaded cluster ID: 23311b9a28e24aef83c1c3eb8b8661c7
I20260927 05:13:43.994511 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:13:43.998224 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:13:43.999596 9223 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 630105165248458abfb6ee39b4bb6afe: Loaded TSK: 0
I20260927 05:13:44.001026 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:13:44.259773 9216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:44.260059 9216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:44.260214 9216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:44.267185 9216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:44.267305 9216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:44.267477 9216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:13:44.277209 9216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34557
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:44.278468 9216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:44.279838 9216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:44.292786 9241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:44.292883 9240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:44.293380 9243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:44.293583 9216 server_base.cc:1034] running on GCE node
I20260927 05:13:44.294202 9216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:44.295176 9216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:44.296398 9216 hybrid_clock.cc:648] HybridClock initialized: now 1790486024296338 us; error 40 us; skew 500 ppm
I20260927 05:13:44.296690 9216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486024, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:44.299793 9216 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:13:44.301287 9216 webserver.cc:460] Webserver started at http://127.0.97.1:34557/ using document root <none> and password file <none>
I20260927 05:13:44.301921 9216 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:44.302073 9216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:44.307204 9216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:13:44.309840 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.311131 9216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:44.311286 9216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "18d7d164e7884e3891d7b73032ae84f0"
format_stamp: "Formatted at 2026-09-27 05:13:41 on dist-test-slave-1t6p"
server_key: "ccf63ad027f3e33244878c2e94081b2b"
server_key_iv: "f59d090d2dd11cf88b88930a08ca1eeb"
server_key_version: "encryptionkey@0"
I20260927 05:13:44.312220 9216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:44.339450 9216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:44.343380 9216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:44.343645 9216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:44.344586 9216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:44.345849 9216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:44.345968 9216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.346093 9216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:44.346177 9216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.377588 9216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39547
I20260927 05:13:44.377671 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39547 every 8 connection(s)
I20260927 05:13:44.379155 9216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:13:44.386514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9216
I20260927 05:13:44.387140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34083
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44777
--webserver_port=43719
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:44.397832 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.382160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45043 (local address 127.0.97.62:44777)
0927 05:13:44.382887 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.382902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.383321 (+ 419us) server_negotiation.cc:407] Connection header received
0927 05:13:44.385213 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.385223 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.385513 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.385849 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.387534 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.389278 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.390791 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.391482 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.391666 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.397026 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45043: BlockingRecv error: recv got EOF from 127.0.97.1:45043 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":249,"threads_started":1}
I20260927 05:13:44.397899 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.382253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:45043)
0927 05:13:44.383078 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.383133 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.384720 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.385987 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.386000 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.386666 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.387337 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.387361 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.389494 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.389498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.390650 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.390661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.391697 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.391704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.391961 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.392596 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.392617 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.396722 (+ 4105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
W20260927 05:13:44.398634 9363 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44777 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44777: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:44.407191 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.399231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:42543)
0927 05:13:44.399395 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.399413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.399513 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.399873 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.399876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.400147 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.400264 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.400270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.401751 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.401755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.402695 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.402705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.403759 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.403763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.403876 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.404464 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.404479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.406946 (+ 2467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:13:44.407490 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.399393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42543 (local address 127.0.97.62:44777)
0927 05:13:44.399584 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.399592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.399612 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:13:44.399663 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.399671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.399748 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.399853 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.400391 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.401613 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.402831 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.403622 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.403762 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.407222 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42543: BlockingRecv error: recv got EOF from 127.0.97.1:42543 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:44.414832 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.407952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:37701)
0927 05:13:44.408098 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.408117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.408218 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.408546 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.408551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.408832 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.408956 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.408963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.409861 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.409866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.410748 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.410757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.411490 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.411496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.411651 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.412160 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.412174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.414568 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 05:13:44.415148 9363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:13:44.415354 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.408124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37701 (local address 127.0.97.62:44777)
0927 05:13:44.408292 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.408298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.408313 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:13:44.408358 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.408361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.408419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.408529 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.409078 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.409736 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.410876 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.411355 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.411481 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.414722 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37701: BlockingRecv error: recv got EOF from 127.0.97.1:37701 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 05:13:44.709513 9367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:44.709805 9367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:44.709987 9367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:44.717163 9367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:44.717295 9367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:44.717456 9367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:13:44.726694 9367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43719
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:44.728037 9367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:44.729436 9367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:44.742278 9375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:44.742311 9373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:44.742777 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:44.742900 9367 server_base.cc:1034] running on GCE node
I20260927 05:13:44.743532 9367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:44.744386 9367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:44.745570 9367 hybrid_clock.cc:648] HybridClock initialized: now 1790486024745517 us; error 54 us; skew 500 ppm
I20260927 05:13:44.745817 9367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486024, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:44.748800 9367 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:13:44.750030 9367 webserver.cc:460] Webserver started at http://127.0.97.2:43719/ using document root <none> and password file <none>
I20260927 05:13:44.750618 9367 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:44.750718 9367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:44.755556 9367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:44.758046 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.759323 9367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:13:44.759483 9367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "638b67779a6f4523ae586481f674482a"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "2cc5c724b8d1edef96b4efb0c1d5f25e"
server_key_iv: "f6187661aaa618f997dfde7495d278ae"
server_key_version: "encryptionkey@0"
I20260927 05:13:44.760356 9367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:44.791469 9367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:44.795368 9367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:44.795629 9367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:44.796562 9367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:44.797894 9367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:44.798002 9367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.798134 9367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:44.798228 9367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:44.830300 9367 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34083
I20260927 05:13:44.830385 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34083 every 8 connection(s)
I20260927 05:13:44.831766 9367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:13:44.839766 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9367
I20260927 05:13:44.840454 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37677
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44777
--webserver_port=34423
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:44.848599 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.834172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59749 (local address 127.0.97.62:44777)
0927 05:13:44.834377 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.834384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.835577 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 05:13:44.837252 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.837256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.837322 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.837409 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.838793 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.839544 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.841986 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.842383 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.842476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.848320 (+ 5844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59749: BlockingRecv error: recv got EOF from 127.0.97.2:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:13:44.849056 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.834701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:59749)
0927 05:13:44.835421 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.835456 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.837008 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.837607 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.837614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.838119 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.838645 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.838661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.839697 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.839704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.841843 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.841861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.842603 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.842612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.842916 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.843647 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.843679 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.848066 (+ 4387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
W20260927 05:13:44.849752 9495 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44777 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44777: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:44.858198 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.850314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:56131)
0927 05:13:44.850445 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.850462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.850545 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.850885 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.850888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.851221 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.851346 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.851353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.852773 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.852776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.853629 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.853639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.854615 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.854619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.854734 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.855309 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.855324 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.858013 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 05:13:44.858593 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.850405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56131 (local address 127.0.97.62:44777)
0927 05:13:44.850592 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.850599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.850620 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:13:44.850674 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.850682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.850762 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.850860 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.851473 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.852644 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.853762 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.854488 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.854620 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.858305 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56131: BlockingRecv error: recv got EOF from 127.0.97.2:56131 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:44 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:44.866317 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:44.858867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:58387)
0927 05:13:44.859038 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:44.859057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:44.859153 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:44.859602 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:44.859606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:44.859888 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:44.860030 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.860037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.861063 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.861068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.861946 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:44.861956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.862615 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.862621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:44.862786 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.863383 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:44.863398 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:44.866039 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:13:44.866412 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:44.859216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58387 (local address 127.0.97.62:44777)
0927 05:13:44.859371 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:44.859376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:44.859388 (+ 12us) server_negotiation.cc:407] Connection header received
0927 05:13:44.859424 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:44.859427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:44.859475 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:44.859581 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:44.860164 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.860918 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.862089 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:44.862466 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:44.862596 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:44.866205 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58387: BlockingRecv error: recv got EOF from 127.0.97.2:58387 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 05:13:44.866585 9495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:13:45.163731 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:13:45.163978 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:13:45.164124 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:13:45.171082 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:13:45.171178 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:13:45.171290 9498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:13:45.180613 9498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34423
--tserver_master_addrs=127.0.97.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:13:45.181732 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:13:45.182879 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:13:45.195509 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:45.195629 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:13:45.195672 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:13:45.196481 9498 server_base.cc:1034] running on GCE node
I20260927 05:13:45.197115 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:13:45.197961 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:13:45.199141 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790486025199097 us; error 32 us; skew 500 ppm
I20260927 05:13:45.199438 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486025, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:13:45.205495 9498 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:13:45.206830 9498 webserver.cc:460] Webserver started at http://127.0.97.3:34423/ using document root <none> and password file <none>
I20260927 05:13:45.207494 9498 fs_manager.cc:359] Metadata directory not provided
I20260927 05:13:45.207604 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:13:45.212543 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:13:45.215039 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:45.216302 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:13:45.216475 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "ddf75da8269f4d8d8b2eb4e3620e1722"
format_stamp: "Formatted at 2026-09-27 05:13:42 on dist-test-slave-1t6p"
server_key: "4f4d736b35036d6d99cbec3fce442da1"
server_key_iv: "87585d2ee267d51c599b167e96cd00dd"
server_key_version: "encryptionkey@0"
I20260927 05:13:45.217417 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:13:45.230652 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:13:45.234643 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:13:45.234877 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:13:45.235889 9498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:13:45.237103 9498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:13:45.237218 9498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:45.237336 9498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:13:45.237424 9498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:13:45.269081 9498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37677
I20260927 05:13:45.269158 9625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37677 every 8 connection(s)
I20260927 05:13:45.270735 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:13:45.273476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9498
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:45.297914 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:45.273980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40695 (local address 127.0.97.62:44777)
0927 05:13:45.274165 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.274172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.279789 (+ 5617us) server_negotiation.cc:407] Connection header received
0927 05:13:45.285544 (+ 5755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.285550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.285628 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.285734 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.287462 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.288634 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.290913 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.291468 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.291606 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.297635 (+ 6029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40695: BlockingRecv error: recv got EOF from 127.0.97.3:40695 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:13:45.299502 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.274744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:40695)
0927 05:13:45.279641 (+ 4897us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:45.279687 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:45.285163 (+ 5476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:45.285997 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:45.286005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:45.286535 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:45.287264 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.287292 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.288766 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.288772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.290763 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.290778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.291691 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.291698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.291916 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.292581 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:45.292616 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:45.298326 (+ 5710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4598,"thread_start_us":4305,"threads_started":1}
W20260927 05:13:45.300319 9626 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44777 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44777: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 05:13:45.303279 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.279037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.0.97.62:44777)
0927 05:13:45.279576 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.279617 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.279636 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:13:45.279702 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.279708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.279801 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.279904 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.282368 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.283647 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.287125 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.287905 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.288061 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.294061 (+ 6000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:13:45.294075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:13:45.294097 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:13:45.294142 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:13:45.298435 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:45.299298 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:45.299306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:45.299314 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:45.299514 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:13:45.300505 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:13:45.300510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:13:45.300513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:13:45.301068 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:13:45.301354 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:13:45.302514 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0927 05:13:45.302846 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":1026,"threads_started":1}
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:45.311609 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:45.300992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59295 (local address 127.0.97.62:44777)
0927 05:13:45.301144 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.301150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.301168 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:13:45.301336 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.301340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.301405 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.301526 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.302245 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.303358 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.306173 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.306829 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.306957 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.311364 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59295: BlockingRecv error: recv got EOF from 127.0.97.3:59295 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 05:13:45.312057 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.300918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:59295)
0927 05:13:45.301097 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:45.301117 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:45.301224 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:45.301544 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:45.301548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:45.301941 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:45.302129 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.302139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.304573 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.304579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.306024 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.306036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.306979 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.306985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.307201 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.307806 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:45.307829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:45.311857 (+ 4028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:13:45.320425 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:45.312832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34663 (local address 127.0.97.62:44777)
0927 05:13:45.313104 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.313111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.313130 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:13:45.313184 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.313192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.313268 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.313398 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.313949 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.314655 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.315957 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.316326 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.316531 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.320235 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34663: BlockingRecv error: recv got EOF from 127.0.97.3:34663 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 05:13:45.320837 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.312706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:34663)
0927 05:13:45.312838 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:45.312854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:45.312938 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:45.313440 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:45.313443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:45.313687 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:45.313815 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.313822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.314780 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.314788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.315816 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.315826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.316544 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.316550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.316704 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.317131 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:45.317146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:45.320613 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 05:13:45.321148 9626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:45.423693 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.415701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:59509)
0927 05:13:45.416171 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:45.416186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:45.416290 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:45.416549 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:45.416552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:45.416964 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:45.417180 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.417193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.418189 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.418194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.419225 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.419259 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.419971 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.419976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.420155 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.420739 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:45.420756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:45.423483 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
W20260927 05:13:45.423871 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:45.415735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59509 (local address 127.0.97.62:44777)
0927 05:13:45.415891 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.415896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.416256 (+ 360us) server_negotiation.cc:407] Connection header received
0927 05:13:45.416373 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.416376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.416434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.416509 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.417314 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.417993 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.419403 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.419794 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.419926 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.423681 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59509: BlockingRecv error: recv got EOF from 127.0.97.1:59509 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:45 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:45.874843 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:45.867163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:46929)
0927 05:13:45.867614 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:45.867628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:45.867796 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:45.868103 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:45.868107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:45.868559 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:45.868729 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.868737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.869697 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.869704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.870643 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:45.870652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.871414 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.871420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:45.871585 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.872145 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:45.872160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:45.874642 (+ 2482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
W20260927 05:13:45.875048 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:45.867190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46929 (local address 127.0.97.62:44777)
0927 05:13:45.867393 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:45.867402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:45.867710 (+ 308us) server_negotiation.cc:407] Connection header received
0927 05:13:45.867909 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:45.867915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:45.867981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:45.868074 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:45.868865 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.869565 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.870790 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:45.871250 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:45.871384 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:45.874815 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46929: BlockingRecv error: recv got EOF from 127.0.97.2:46929 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:46.329625 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:46.321716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:38405)
0927 05:13:46.322284 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:46.322299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:46.322454 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:46.322873 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:46.322878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:46.323531 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:46.323775 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.323784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.324722 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.324725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.325622 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.325632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.326294 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.326298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.326421 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.326940 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:46.326956 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:46.329426 (+ 2470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":335,"threads_started":1}
W20260927 05:13:46.329772 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:46.321762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38405 (local address 127.0.97.62:44777)
0927 05:13:46.321937 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:46.321941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:46.322392 (+ 451us) server_negotiation.cc:407] Connection header received
0927 05:13:46.322598 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:46.322603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:46.322682 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:46.322806 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:46.323899 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.324599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.325733 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.326156 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.326243 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.329578 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38405: BlockingRecv error: recv got EOF from 127.0.97.3:38405 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:46.432722 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:46.424625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:47919)
0927 05:13:46.425097 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:46.425112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:46.425263 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:46.425597 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:46.425601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:46.426023 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:46.426157 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.426165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.427223 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.427233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.428291 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.428302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.429205 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.429210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.429315 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.429830 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:46.429846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:46.432499 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
W20260927 05:13:46.432884 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:46.424688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47919 (local address 127.0.97.62:44777)
0927 05:13:46.424915 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:46.424924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:46.425205 (+ 281us) server_negotiation.cc:407] Connection header received
0927 05:13:46.425387 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:46.425392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:46.425460 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:46.425567 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:46.426299 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.427092 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.428483 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.429063 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.429170 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.432651 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47919: BlockingRecv error: recv got EOF from 127.0.97.1:47919 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:46 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:46.889328 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:46.875718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:35227)
0927 05:13:46.876239 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:46.876253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:46.876359 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:46.876636 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:46.876639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:46.876990 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:46.877120 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.877127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.878110 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.878114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.879097 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:46.879108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.879738 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.879747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:46.879899 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.886076 (+ 6177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:46.886095 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:46.889100 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":298,"threads_started":1}
W20260927 05:13:46.889557 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:46.875801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35227 (local address 127.0.97.62:44777)
0927 05:13:46.875955 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:46.875959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:46.876321 (+ 362us) server_negotiation.cc:407] Connection header received
0927 05:13:46.876447 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:46.876451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:46.876521 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:46.876596 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:46.877230 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.877975 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.879200 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:46.879580 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:46.879701 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:46.889295 (+ 9594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35227: BlockingRecv error: recv got EOF from 127.0.97.2:35227 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:47.338577 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:47.330541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:47443)
0927 05:13:47.331095 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:47.331115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:47.331266 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:47.331585 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:47.331589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:47.331993 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:47.332139 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.332147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.333127 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.333132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.334056 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.334067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.334773 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.334778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.334943 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.335598 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:47.335616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:47.338243 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":276,"threads_started":1}
W20260927 05:13:47.338660 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:47.330863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47443 (local address 127.0.97.62:44777)
0927 05:13:47.331095 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:47.331102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:47.331192 (+ 90us) server_negotiation.cc:407] Connection header received
0927 05:13:47.331394 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:47.331403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:47.331460 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:47.331558 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:47.332296 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.332985 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.334195 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.334618 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.334743 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.338417 (+ 3674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47443: BlockingRecv error: recv got EOF from 127.0.97.3:47443 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:47.441363 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:47.433675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:52853)
0927 05:13:47.434377 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:47.434393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:47.434514 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:47.434765 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:47.434768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:47.435121 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:47.435305 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.435313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.436262 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.436266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.437144 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.437154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.437738 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.437742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.437853 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.438488 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:47.438505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:47.441130 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":472,"threads_started":1}
W20260927 05:13:47.441497 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:47.433715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52853 (local address 127.0.97.62:44777)
0927 05:13:47.433880 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:47.433885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:47.434466 (+ 581us) server_negotiation.cc:407] Connection header received
0927 05:13:47.434612 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:47.434616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:47.434686 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:47.434751 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:47.435423 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.436150 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.437255 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.437631 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.437714 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.441297 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52853: BlockingRecv error: recv got EOF from 127.0.97.1:52853 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:47 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:47.898267 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:47.890193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:53911)
0927 05:13:47.890733 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:47.890749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:47.890898 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:47.891209 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:47.891212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:47.891526 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:47.891692 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.891700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.892742 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.892747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.893787 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:47.893801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.894465 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.894474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:47.894600 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.895266 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:47.895283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:47.898007 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":302,"threads_started":1}
W20260927 05:13:47.898396 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:47.890293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53911 (local address 127.0.97.62:44777)
0927 05:13:47.890483 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:47.890488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:47.890852 (+ 364us) server_negotiation.cc:407] Connection header received
0927 05:13:47.891018 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:47.891025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:47.891086 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:47.891165 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:47.891818 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.892580 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.893931 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:47.894313 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:47.894430 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:47.898178 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53911: BlockingRecv error: recv got EOF from 127.0.97.2:53911 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:48.347811 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:48.339658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:51437)
0927 05:13:48.340214 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:48.340229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:48.340338 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:48.340638 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:48.340641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:48.340944 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:48.341114 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.341121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.342167 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.342170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.343048 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.343058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.343730 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.343735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.343856 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.344606 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:48.344624 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:48.347571 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":342,"threads_started":1}
W20260927 05:13:48.347966 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:48.339734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51437 (local address 127.0.97.62:44777)
0927 05:13:48.339884 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:48.339889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:48.340315 (+ 426us) server_negotiation.cc:407] Connection header received
0927 05:13:48.340439 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:48.340442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:48.340515 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:48.340600 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:48.341245 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.342048 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.343178 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.343565 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.343662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.347762 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51437: BlockingRecv error: recv got EOF from 127.0.97.3:51437 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:48.449702 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:48.442202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:54643)
0927 05:13:48.442649 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:48.442665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:48.442771 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:48.443057 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:48.443060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:48.443394 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:48.443548 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.443556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.444459 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.444462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.445393 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.445403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.446031 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.446035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.446156 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.446689 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:48.446706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:48.449411 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":259,"threads_started":1}
W20260927 05:13:48.449909 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:48.442288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54643 (local address 127.0.97.62:44777)
0927 05:13:48.442555 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:48.442560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:48.442740 (+ 180us) server_negotiation.cc:407] Connection header received
0927 05:13:48.442881 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:48.442884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:48.442942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:48.443034 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:48.443661 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.444358 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.445512 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.445917 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.446013 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.449621 (+ 3608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54643: BlockingRecv error: recv got EOF from 127.0.97.1:54643 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:48 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:48.907081 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:48.899146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:33793)
0927 05:13:48.899615 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:48.899630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:48.899784 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:48.900119 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:48.900123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:48.900502 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:48.900653 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.900661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.901660 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.901667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.902608 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:48.902619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.903362 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.903367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:48.903558 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.904093 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:48.904109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:48.906738 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
W20260927 05:13:48.907197 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:48.899220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33793 (local address 127.0.97.62:44777)
0927 05:13:48.899460 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:48.899469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:48.899713 (+ 244us) server_negotiation.cc:407] Connection header received
0927 05:13:48.899902 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:48.899924 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:48.899992 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:48.900093 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:48.900785 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.901534 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.902756 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:48.903225 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:48.903353 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:48.906913 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33793: BlockingRecv error: recv got EOF from 127.0.97.2:33793 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:49.356760 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:49.348784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:46971)
0927 05:13:49.349234 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:49.349249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:49.349358 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:49.349661 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:49.349663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:49.350073 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:49.350220 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.350228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.351509 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.351512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.352436 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.352446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.353129 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.353133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.353256 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.353870 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:49.353900 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:49.356547 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
W20260927 05:13:49.356973 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:49.348822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46971 (local address 127.0.97.62:44777)
0927 05:13:49.349016 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:49.349025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:49.349347 (+ 322us) server_negotiation.cc:407] Connection header received
0927 05:13:49.349448 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:49.349452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:49.349512 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:49.349649 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:49.350354 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.351380 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.352563 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.353019 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.353109 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.356726 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46971: BlockingRecv error: recv got EOF from 127.0.97.3:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:49.458681 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:49.450700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:49321)
0927 05:13:49.451244 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:49.451259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:49.451411 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:49.451737 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:49.451741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:49.452135 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:49.452292 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.452300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.453308 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.453316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.454231 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.454242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.454936 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.454942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.455173 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.455842 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:49.455859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:49.458349 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
W20260927 05:13:49.458748 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:49.450752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49321 (local address 127.0.97.62:44777)
0927 05:13:49.450995 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:49.451003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:49.451343 (+ 340us) server_negotiation.cc:407] Connection header received
0927 05:13:49.451539 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:49.451544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:49.451609 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:49.451708 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:49.452462 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.453172 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.454364 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.454769 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.454923 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.458539 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49321: BlockingRecv error: recv got EOF from 127.0.97.1:49321 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:49 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:49.916456 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:49.907914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:56057)
0927 05:13:49.908605 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:49.908620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:49.908725 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:49.909007 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:49.909010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:49.909354 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:49.909507 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.909514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.910533 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.910537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.911689 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:49.911703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.912513 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.912519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:49.912691 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.913309 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:49.913325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:49.916157 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":434,"threads_started":1}
W20260927 05:13:49.916558 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:49.908164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56057 (local address 127.0.97.62:44777)
0927 05:13:49.908347 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:49.908351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:49.908694 (+ 343us) server_negotiation.cc:407] Connection header received
0927 05:13:49.908817 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:49.908820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:49.908884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:49.908966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:49.909632 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.910403 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.911892 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:49.912363 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:49.912505 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:49.916308 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56057: BlockingRecv error: recv got EOF from 127.0.97.2:56057 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:50.365131 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:50.357619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:49651)
0927 05:13:50.358104 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:50.358119 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:50.358219 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:50.358494 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:50.358497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:50.358840 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:50.358990 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.359049 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.359974 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.359977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.360889 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.360899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.361501 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.361505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.361614 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.362114 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:50.362131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:50.364903 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
W20260927 05:13:50.365262 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:50.357701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49651 (local address 127.0.97.62:44777)
0927 05:13:50.357860 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:50.357864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:50.358188 (+ 324us) server_negotiation.cc:407] Connection header received
0927 05:13:50.358297 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:50.358301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:50.358357 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:50.358435 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:50.359176 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.359852 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.361002 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.361382 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.361467 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.365060 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49651: BlockingRecv error: recv got EOF from 127.0.97.3:49651 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:50.467873 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:50.459670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:46041)
0927 05:13:50.460161 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:50.460175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:50.460321 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:50.460715 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:50.460719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:50.461128 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:50.461368 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.461378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.462522 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.462525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.463563 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.463575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.464227 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.464232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.464346 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.464918 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:50.464934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:50.467674 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
W20260927 05:13:50.468053 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:50.459741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46041 (local address 127.0.97.62:44777)
0927 05:13:50.459956 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:50.459964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:50.460256 (+ 292us) server_negotiation.cc:407] Connection header received
0927 05:13:50.460462 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:50.460466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:50.460557 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:50.460651 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:50.461508 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.462394 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.463678 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.464063 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.464215 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.467823 (+ 3608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46041: BlockingRecv error: recv got EOF from 127.0.97.1:46041 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:50 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:50.925350 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:50.917340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:41057)
0927 05:13:50.917999 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:50.918014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:50.918163 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:50.918495 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:50.918499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:50.918884 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:50.919058 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.919067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.920115 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.920122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.921089 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:50.921100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.921766 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.921772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:50.921942 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.922465 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:50.922493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:50.925050 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":457,"threads_started":1}
W20260927 05:13:50.925501 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:50.917447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41057 (local address 127.0.97.62:44777)
0927 05:13:50.917639 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:50.917647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:50.918104 (+ 457us) server_negotiation.cc:407] Connection header received
0927 05:13:50.918280 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:50.918286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:50.918357 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:50.918458 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:50.919204 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.919990 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.921221 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:50.921617 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:50.921755 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:50.925267 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41057: BlockingRecv error: recv got EOF from 127.0.97.2:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:51.373420 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:51.366021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:42221)
0927 05:13:51.366489 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:51.366504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:51.366613 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:51.366901 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:51.366903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:51.367291 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:51.367429 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.367437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.368458 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.368461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.369370 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.369380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.370049 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.370052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.370165 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.370709 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:51.370725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:51.373202 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":284,"threads_started":1}
W20260927 05:13:51.373692 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:51.366122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42221 (local address 127.0.97.62:44777)
0927 05:13:51.366352 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:51.366359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:51.366579 (+ 220us) server_negotiation.cc:407] Connection header received
0927 05:13:51.366720 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:51.366724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:51.366783 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:51.366869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:51.367554 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.368347 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.369485 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.369913 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.370006 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.373416 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42221: BlockingRecv error: recv got EOF from 127.0.97.3:42221 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:51.477227 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:51.468838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:60255)
0927 05:13:51.469589 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:51.469611 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:51.469778 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:51.470143 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:51.470148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:51.470587 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:51.470750 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.470758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.471790 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.471796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.472701 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.472712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.473425 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.473431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.473600 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.474292 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:51.474309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:51.476996 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
W20260927 05:13:51.477427 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:51.468882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60255 (local address 127.0.97.62:44777)
0927 05:13:51.469155 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:51.469164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:51.469676 (+ 512us) server_negotiation.cc:407] Connection header received
0927 05:13:51.469940 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:51.469947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:51.470009 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:51.470106 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:51.470888 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.471644 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.472846 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.473267 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.473402 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.477177 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60255: BlockingRecv error: recv got EOF from 127.0.97.1:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:51 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:51.939221 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:51.926307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:58835)
0927 05:13:51.926883 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:51.926898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:51.927076 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:51.927415 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:51.927420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:51.927967 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:51.928207 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.928220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.929358 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.929365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.930248 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:51.930258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.930944 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.930949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:51.931148 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.936189 (+ 5041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:51.936206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:51.938849 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
W20260927 05:13:51.939304 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:51.926400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58835 (local address 127.0.97.62:44777)
0927 05:13:51.926618 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:51.926627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:51.927020 (+ 393us) server_negotiation.cc:407] Connection header received
0927 05:13:51.927216 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:51.927220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:51.927279 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:51.927379 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:51.928353 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.929232 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.930383 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:51.930790 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:51.930916 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:51.939062 (+ 8146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58835: BlockingRecv error: recv got EOF from 127.0.97.2:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:52.382722 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:52.374269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:48679)
0927 05:13:52.374966 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:52.374980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:52.375147 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:52.375462 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:52.375466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:52.375864 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:52.376037 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.376045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.377026 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.377034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.377920 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.377932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.378758 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.378764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.378921 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.379552 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:52.379569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:52.382411 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":373,"threads_started":1}
W20260927 05:13:52.382874 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:52.374368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48679 (local address 127.0.97.62:44777)
0927 05:13:52.374675 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:52.374683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:52.375080 (+ 397us) server_negotiation.cc:407] Connection header received
0927 05:13:52.375261 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:52.375266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:52.375337 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:52.375438 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:52.376170 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.376898 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.378071 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.378614 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.378733 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.382653 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48679: BlockingRecv error: recv got EOF from 127.0.97.3:48679 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:52.486107 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:52.478122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:56897)
0927 05:13:52.478587 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:52.478606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:52.478723 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:52.479077 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:52.479082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:52.479572 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:52.479733 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.479741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.480857 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.480863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.481785 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.481796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.482533 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.482540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.482693 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.483285 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:52.483302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:52.485888 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
W20260927 05:13:52.486358 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:52.478188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56897 (local address 127.0.97.62:44777)
0927 05:13:52.478400 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:52.478406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:52.478691 (+ 285us) server_negotiation.cc:407] Connection header received
0927 05:13:52.478817 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:52.478820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:52.478886 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:52.478959 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:52.479886 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.480693 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.481929 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.482356 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.482492 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.486083 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56897: BlockingRecv error: recv got EOF from 127.0.97.1:56897 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:52 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:52.948380 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:52.940199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:49959)
0927 05:13:52.940637 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:52.940652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:52.940761 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:52.941069 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:52.941073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:52.941437 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:52.941584 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.941591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.943108 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.943114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.944037 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:52.944047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.944717 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.944721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:52.944826 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.945455 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:52.945496 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:52.948174 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
W20260927 05:13:52.948540 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:52.940341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49959 (local address 127.0.97.62:44777)
0927 05:13:52.940565 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:52.940573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:52.940725 (+ 152us) server_negotiation.cc:407] Connection header received
0927 05:13:52.940857 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:52.940860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:52.940917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:52.940994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:52.941736 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.942933 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.944156 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:52.944571 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:52.944658 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:52.948339 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49959: BlockingRecv error: recv got EOF from 127.0.97.2:49959 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:53.391952 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:53.383787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:32937)
0927 05:13:53.384404 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:53.384419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:53.384558 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:53.384891 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:53.384896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:53.385308 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:53.385456 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.385464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.386462 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.386468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.387492 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.387503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.388251 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.388256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.388428 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.388968 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:53.388984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:53.391629 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":402,"threads_started":1}
W20260927 05:13:53.392086 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:53.383836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32937 (local address 127.0.97.62:44777)
0927 05:13:53.384042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:53.384048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:53.384495 (+ 447us) server_negotiation.cc:407] Connection header received
0927 05:13:53.384688 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:53.384697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:53.384753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:53.384864 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:53.385588 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.386320 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.387634 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.388057 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.388195 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.391816 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:32937: BlockingRecv error: recv got EOF from 127.0.97.3:32937 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:53.494954 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:53.486900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:38519)
0927 05:13:53.487445 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:53.487461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:53.487610 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:53.487997 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:53.488003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:53.488490 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:53.488708 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.488724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.489797 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.489800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.490701 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.490710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.491359 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.491364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.491485 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.492092 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:53.492109 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:53.494724 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":332,"threads_started":1}
W20260927 05:13:53.495136 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:53.486981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38519 (local address 127.0.97.62:44777)
0927 05:13:53.487168 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:53.487173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:53.487542 (+ 369us) server_negotiation.cc:407] Connection header received
0927 05:13:53.487768 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:53.487772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:53.487841 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:53.487918 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:53.488857 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.489677 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.490819 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.491241 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.491358 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.494894 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38519: BlockingRecv error: recv got EOF from 127.0.97.1:38519 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:53 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:53.957513 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:53.949451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:35165)
0927 05:13:53.949925 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:53.949941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:53.950060 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:53.950329 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:53.950332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:53.950681 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:53.950846 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.950853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.952021 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.952025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.952919 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:53.952929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.953780 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.953787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:53.953960 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.954647 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:53.954669 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:53.957276 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":272,"threads_started":1}
W20260927 05:13:53.957662 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:53.949506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35165 (local address 127.0.97.62:44777)
0927 05:13:53.949707 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:53.949716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:53.950023 (+ 307us) server_negotiation.cc:407] Connection header received
0927 05:13:53.950149 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:53.950152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:53.950211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:53.950287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:53.950971 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.951897 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.953051 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:53.953599 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:53.953732 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:53.957466 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35165: BlockingRecv error: recv got EOF from 127.0.97.2:35165 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:54.400729 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:54.392841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:56865)
0927 05:13:54.393335 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:54.393352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:54.393458 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:54.393738 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:54.393741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:54.394138 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:54.394298 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.394307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.395273 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.395276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.396286 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.396297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.397131 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.397136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.397318 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.397847 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:54.397862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:54.400488 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
W20260927 05:13:54.400885 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:54.392867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56865 (local address 127.0.97.62:44777)
0927 05:13:54.393068 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:54.393077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:54.393426 (+ 349us) server_negotiation.cc:407] Connection header received
0927 05:13:54.393551 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:54.393555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:54.393613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:54.393700 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:54.394420 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.395165 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.396416 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.396990 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.397119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.400670 (+ 3551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56865: BlockingRecv error: recv got EOF from 127.0.97.3:56865 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:54.504400 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:54.496033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:40151)
0927 05:13:54.496505 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:54.496521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:54.496629 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:54.496966 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:54.496969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:54.497364 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:54.497502 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.497510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.498476 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.498480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.499517 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.499529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.500182 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.500186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.500299 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.500933 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:54.500950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:54.504186 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":286,"threads_started":1}
W20260927 05:13:54.504598 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:54.496112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40151 (local address 127.0.97.62:44777)
0927 05:13:54.496356 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:54.496362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:54.496601 (+ 239us) server_negotiation.cc:407] Connection header received
0927 05:13:54.496730 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:54.496734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:54.496808 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:54.496888 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:54.497625 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.498345 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.499639 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.500057 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.500151 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.504359 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40151: BlockingRecv error: recv got EOF from 127.0.97.1:40151 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:54 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:54.966912 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:54.958404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:50059)
0927 05:13:54.958894 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:54.958910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:54.959083 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:54.959429 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:54.959432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:54.959898 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:54.960085 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.960100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.961295 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.961301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.962210 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:54.962221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.962917 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.962923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:54.963117 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.963760 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:54.963776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:54.966513 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":291,"threads_started":1}
W20260927 05:13:54.966979 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:54.958508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50059 (local address 127.0.97.62:44777)
0927 05:13:54.958667 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:54.958672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:54.959045 (+ 373us) server_negotiation.cc:407] Connection header received
0927 05:13:54.959202 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:54.959205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:54.959299 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:54.959411 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:54.960252 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.961161 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.962348 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:54.962757 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:54.962890 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:54.966736 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50059: BlockingRecv error: recv got EOF from 127.0.97.2:50059 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:55.410069 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:55.401567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:46915)
0927 05:13:55.402118 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:55.402138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:55.402312 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:55.402650 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:55.402654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:55.403073 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:55.403213 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.403222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.404236 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.404241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.405162 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.405172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.405902 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.405908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.406091 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.406821 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:55.406846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:55.409860 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
W20260927 05:13:55.410336 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:55.401872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46915 (local address 127.0.97.62:44777)
0927 05:13:55.402119 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:55.402126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:55.402236 (+ 110us) server_negotiation.cc:407] Connection header received
0927 05:13:55.402434 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:55.402441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:55.402514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:55.402616 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:55.403358 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.404092 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.405299 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.405730 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.405884 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.410095 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46915: BlockingRecv error: recv got EOF from 127.0.97.3:46915 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:55.513231 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:55.505247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:41193)
0927 05:13:55.505766 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:55.505797 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:55.505913 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:55.506206 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:55.506209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:55.506607 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:55.506769 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.506777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.507763 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.507770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.508663 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.508674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.509325 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.509329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.509434 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.509991 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:55.510007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:55.512888 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":293,"threads_started":1}
W20260927 05:13:55.513305 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:55.505503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41193 (local address 127.0.97.62:44777)
0927 05:13:55.505717 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:55.505724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:55.505879 (+ 155us) server_negotiation.cc:407] Connection header received
0927 05:13:55.506012 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:55.506015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:55.506086 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:55.506163 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:55.506903 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.507626 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.508783 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.509205 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.509294 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.513065 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41193: BlockingRecv error: recv got EOF from 127.0.97.1:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:55 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:55.976087 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:55.967928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:47643)
0927 05:13:55.968418 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:55.968434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:55.968546 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:55.968819 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:55.968822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:55.969278 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:55.969498 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.969507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.970685 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.970691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.971694 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:55.971705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.972406 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.972410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:55.972522 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.973080 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:55.973096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:55.975849 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":277,"threads_started":1}
W20260927 05:13:55.976257 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:55.967991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47643 (local address 127.0.97.62:44777)
0927 05:13:55.968167 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:55.968172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:55.968505 (+ 333us) server_negotiation.cc:407] Connection header received
0927 05:13:55.968638 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:55.968642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:55.968702 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:55.968781 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:55.969621 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.970544 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.971816 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:55.972278 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:55.972376 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:55.976024 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47643: BlockingRecv error: recv got EOF from 127.0.97.2:47643 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:56.419133 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:56.410958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:53335)
0927 05:13:56.411513 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:56.411529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:56.411631 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:56.411923 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:56.411926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:56.412333 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:56.412498 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.412506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.413451 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.413454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.414371 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.414382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.414990 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.414994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.415118 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.415772 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:56.415789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:56.418847 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
W20260927 05:13:56.419232 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:56.411075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53335 (local address 127.0.97.62:44777)
0927 05:13:56.411244 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:56.411251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:56.411599 (+ 348us) server_negotiation.cc:407] Connection header received
0927 05:13:56.411718 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:56.411722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:56.411783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:56.411892 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:56.412606 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.413348 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.414486 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.414872 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.414957 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.418986 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53335: BlockingRecv error: recv got EOF from 127.0.97.3:53335 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:56.522557 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:56.514122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:37357)
0927 05:13:56.514600 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:56.514616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:56.514773 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:56.515164 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:56.515168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:56.515575 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:56.515729 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.515737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.516793 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.516800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.517698 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.517708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.518447 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.518453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.518625 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.519262 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:56.519279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:56.522223 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":268,"threads_started":1}
W20260927 05:13:56.522624 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:56.514184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37357 (local address 127.0.97.62:44777)
0927 05:13:56.514418 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:56.514426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:56.514720 (+ 294us) server_negotiation.cc:407] Connection header received
0927 05:13:56.514902 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:56.514907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:56.514990 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:56.515138 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:56.515902 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.516652 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.517834 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.518300 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.518425 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.522402 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37357: BlockingRecv error: recv got EOF from 127.0.97.1:37357 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:56 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:56.990315 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:56.977000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:58061)
0927 05:13:56.977440 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:56.977455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:56.977561 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:56.977818 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:56.977821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:56.978249 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:56.978399 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.978406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.979405 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.979410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.980517 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:56.980528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.981264 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.981269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:56.981401 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.986918 (+ 5517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:56.986935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:56.990095 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
W20260927 05:13:56.990513 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:56.977007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58061 (local address 127.0.97.62:44777)
0927 05:13:56.977220 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:56.977229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:56.977529 (+ 300us) server_negotiation.cc:407] Connection header received
0927 05:13:56.977652 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:56.977656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:56.977714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:56.977789 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:56.978508 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.979280 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.980634 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:56.981122 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:56.981213 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:56.990296 (+ 9083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58061: BlockingRecv error: recv got EOF from 127.0.97.2:58061 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:57.428920 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:57.420110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:42421)
0927 05:13:57.420613 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:57.420630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:57.420738 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:57.421116 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:57.421121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:57.421722 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:57.421985 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.422000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.423313 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.423317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.424229 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.424240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.425254 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.425263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.425442 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.426055 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:57.426071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:57.428683 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":313,"threads_started":1}
W20260927 05:13:57.429090 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:57.420156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42421 (local address 127.0.97.62:44777)
0927 05:13:57.420446 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:57.420452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:57.420705 (+ 253us) server_negotiation.cc:407] Connection header received
0927 05:13:57.420859 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:57.420863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:57.420921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:57.421009 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:57.422144 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.423189 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.424362 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.425080 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.425215 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.428857 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42421: BlockingRecv error: recv got EOF from 127.0.97.3:42421 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:57.531903 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:57.523458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:45435)
0927 05:13:57.523972 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:57.523987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:57.524137 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:57.524495 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:57.524500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:57.524883 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:57.525043 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.525052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.526065 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.526072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.527230 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.527242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.528147 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.528153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.528313 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.528943 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:57.528967 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:57.531683 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":255,"threads_started":1}
W20260927 05:13:57.532220 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:57.523523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45435 (local address 127.0.97.62:44777)
0927 05:13:57.523750 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:57.523758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:57.524067 (+ 309us) server_negotiation.cc:407] Connection header received
0927 05:13:57.524276 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:57.524280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:57.524360 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:57.524466 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:57.525181 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.525926 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.527363 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.527968 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.528106 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.531843 (+ 3737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45435: BlockingRecv error: recv got EOF from 127.0.97.1:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:57 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:57.999317 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:57.991272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:50025)
0927 05:13:57.991791 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:57.991807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:57.991926 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:57.992199 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:57.992203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:57.992560 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:57.992706 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.992714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.993714 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.993718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.995171 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:57.995191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.995847 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.995853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:57.995962 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.996504 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:57.996520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:57.999103 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
W20260927 05:13:57.999506 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:57.991534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50025 (local address 127.0.97.62:44777)
0927 05:13:57.991706 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:57.991710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:57.991876 (+ 166us) server_negotiation.cc:407] Connection header received
0927 05:13:57.992023 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:57.992026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:57.992082 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:57.992160 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:57.992818 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.993563 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.995298 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:57.995700 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:57.995792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:57.999266 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50025: BlockingRecv error: recv got EOF from 127.0.97.2:50025 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 05:13:58 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:58 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:58.437974 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:58.429826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:43775)
0927 05:13:58.430327 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:58.430341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:58.430499 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:58.430814 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:58.430820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:58.431302 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:58.431452 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:58.431460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.432458 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.432465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:58.433393 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:58.433404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.434093 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.434099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:58.434298 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:58.434919 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:58.434935 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:58.437668 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
W20260927 05:13:58.438082 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:58.429866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43775 (local address 127.0.97.62:44777)
0927 05:13:58.430068 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:58.430076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:58.430431 (+ 355us) server_negotiation.cc:407] Connection header received
0927 05:13:58.430619 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:58.430625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:58.430692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:58.430785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:58.431591 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.432327 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.433532 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.433928 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.434060 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:58.437860 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43775: BlockingRecv error: recv got EOF from 127.0.97.3:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 05:13:58 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:58 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:58.540639 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:58.532660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:42077)
0927 05:13:58.533119 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:58.533134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:58.533274 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:58.533682 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:58.533689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:58.534077 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:58.534246 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:58.534255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.535336 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.535341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:58.536254 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:58.536264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.536942 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.536957 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:58.537131 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:58.537762 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:58.537779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:58.540401 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
W20260927 05:13:58.540789 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:58.532721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42077 (local address 127.0.97.62:44777)
0927 05:13:58.532983 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:58.532989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:58.533206 (+ 217us) server_negotiation.cc:407] Connection header received
0927 05:13:58.533413 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:58.533420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:58.533526 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:58.533645 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:58.534383 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.535199 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.536394 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:58.536804 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:58.536932 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:58.540580 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42077: BlockingRecv error: recv got EOF from 127.0.97.1:42077 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:59.008621 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:59.000366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:40401)
0927 05:13:59.000884 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:59.000904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:59.001043 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:59.001428 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:59.001434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:59.001812 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:59.001974 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.001983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.002971 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.002977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.003895 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.003905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.004610 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.004615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.004793 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.005394 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:59.005414 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:59.008378 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":245,"threads_started":1}
W20260927 05:13:59.008769 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:59.000458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40401 (local address 127.0.97.62:44777)
0927 05:13:59.000646 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:59.000653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:59.000972 (+ 319us) server_negotiation.cc:407] Connection header received
0927 05:13:59.001164 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:59.001170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:59.001243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:59.001377 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:59.002115 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.002827 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.004030 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.004471 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.004600 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.008566 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40401: BlockingRecv error: recv got EOF from 127.0.97.2:40401 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:59.447219 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:59.439113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.3:44009)
0927 05:13:59.439670 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:59.439687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:59.439804 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:59.440088 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:59.440091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:59.440479 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:59.440648 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.440657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.441549 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.441553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.442572 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.442584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.443294 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.443298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.443444 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.444143 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:59.444160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:59.446915 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":315,"threads_started":1}
W20260927 05:13:59.447376 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:59.439139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44009 (local address 127.0.97.62:44777)
0927 05:13:59.439297 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:59.439302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:59.439759 (+ 457us) server_negotiation.cc:407] Connection header received
0927 05:13:59.439897 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:59.439900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:59.439965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:59.440044 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:59.440761 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.441419 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.442690 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.443157 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.443245 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.447186 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44009: BlockingRecv error: recv got EOF from 127.0.97.3:44009 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:13:59 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:13:59.549584 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:13:59.541682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.1:47565)
0927 05:13:59.542172 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 05:13:59.542187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:13:59.542341 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:13:59.542669 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:13:59.542673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:13:59.543098 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:13:59.543294 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.543301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.544293 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.544299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.545191 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:13:59.545201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.545939 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.545945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:13:59.546119 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.546677 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:13:59.546692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:13:59.549318 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":275,"threads_started":1}
W20260927 05:13:59.549937 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:13:59.541758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47565 (local address 127.0.97.62:44777)
0927 05:13:59.542016 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:13:59.542023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:13:59.542270 (+ 247us) server_negotiation.cc:407] Connection header received
0927 05:13:59.542467 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:13:59.542476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:13:59.542546 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:13:59.542645 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:13:59.543442 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.544159 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.545323 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:13:59.545777 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:13:59.545920 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:13:59.549610 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47565: BlockingRecv error: recv got EOF from 127.0.97.1:47565 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 05:14:00 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:14:00 dist-test-slave-1t6p krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:14:00.018000 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:00.009742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44777 (local address 127.0.97.2:45001)
0927 05:14:00.010210 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:00.010225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:00.010379 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:00.010700 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:00.010705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:00.011156 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:00.011339 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:00.011357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.012362 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.012367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:00.013260 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:00.013270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.014009 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.014015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:00.014181 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.014846 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:00.014862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:00.017593 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44777: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
W20260927 05:14:00.018119 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:14:00.009823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45001 (local address 127.0.97.62:44777)
0927 05:14:00.010037 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:00.010045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:00.010307 (+ 262us) server_negotiation.cc:407] Connection header received
0927 05:14:00.010507 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:00.010511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:00.010574 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:00.010672 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:00.011487 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.012214 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.013401 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.013856 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.013989 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.017833 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45001: BlockingRecv error: recv got EOF from 127.0.97.2:45001 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:14:00.274237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9216
I20260927 05:14:00.285688 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:00.275207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.97.1:39547)
0927 05:14:00.275736 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:00.275743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:00.275768 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:14:00.275819 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:00.275824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:00.276035 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:00.276189 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:00.277581 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.278724 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.279993 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.280507 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.280603 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.283070 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:00.283078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:00.283092 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:00.283128 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:00.285248 (+ 2120us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:14:00.285472 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42118: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":291,"threads_started":1}
W20260927 05:14:00.285809 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42118: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:14:00.286057 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9216: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39547: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:14:00.286255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9216
I20260927 05:14:00.302637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9367
I20260927 05:14:00.313860 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:00.303600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.0.97.2:34083)
0927 05:14:00.304226 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:00.304233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:00.304261 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:14:00.304314 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:00.304321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:00.304523 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:00.304670 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:00.305797 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.306862 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.308076 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.308595 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.308689 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.310856 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:00.310864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:00.310877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:00.310908 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:00.313374 (+ 2466us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:14:00.313580 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41004: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":240,"threads_started":1}
W20260927 05:14:00.313903 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9367: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34083: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:14:00.313958 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41004: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:14:00.314081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9367
I20260927 05:14:00.330049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9498
I20260927 05:14:00.341217 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:00.330889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.0.97.3:37677)
0927 05:14:00.331451 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:00.331459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:00.331483 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:14:00.331538 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:00.331544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:00.331752 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:00.331899 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:00.333073 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.334060 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.335295 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.335827 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.335921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.338168 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:00.338176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:00.338188 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:00.338222 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:00.340707 (+ 2485us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:14:00.340942 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45590: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
W20260927 05:14:00.341300 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9498: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37677: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:14:00.341317 9678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45590: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:14:00.341439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9498
I20260927 05:14:00.356751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9147
I20260927 05:14:00.357838 9209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:00.468374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9147
2026-09-27T05:14:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20800 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T05:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:14:00Z Disabled control of system clock
I20260927 05:14:00.541553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34373
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 05:14:00.852409 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:00.852662 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:00.852798 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:00.859819 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:14:00.859913 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:00.859984 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:00.860085 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:14:00.860146 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:14:00.868903 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:00.870098 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:00.871843 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:00.883208 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:00.883237 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:00.883173 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:00.883842 9683 server_base.cc:1034] running on GCE node
I20260927 05:14:00.884598 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:00.885905 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:00.887204 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790486040887155 us; error 24 us; skew 500 ppm
I20260927 05:14:00.887573 9683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:00.889881 9683 webserver.cc:460] Webserver started at http://127.0.97.62:35873/ using document root <none> and password file <none>
I20260927 05:14:00.890854 9683 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:00.891038 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:00.891372 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:00.894188 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "ad973af6b3ca4276a94c970628c5433b"
format_stamp: "Formatted at 2026-09-27 05:14:00 on dist-test-slave-1t6p"
server_key: "7f5e3d52fabee754336059d4c240fc6d"
server_key_iv: "00b156fc58ec9f8d0f876b15e454d2b7"
server_key_version: "encryptionkey@0"
I20260927 05:14:00.894948 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad973af6b3ca4276a94c970628c5433b"
format_stamp: "Formatted at 2026-09-27 05:14:00 on dist-test-slave-1t6p"
server_key: "7f5e3d52fabee754336059d4c240fc6d"
server_key_iv: "00b156fc58ec9f8d0f876b15e454d2b7"
server_key_version: "encryptionkey@0"
I20260927 05:14:00.899807 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:14:00.903080 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:00.904330 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:14:00.904505 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "ad973af6b3ca4276a94c970628c5433b"
format_stamp: "Formatted at 2026-09-27 05:14:00 on dist-test-slave-1t6p"
server_key: "7f5e3d52fabee754336059d4c240fc6d"
server_key_iv: "00b156fc58ec9f8d0f876b15e454d2b7"
server_key_version: "encryptionkey@0"
I20260927 05:14:00.904709 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:00.928346 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:00.929106 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:00.929390 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:00.947425 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34373
I20260927 05:14:00.947454 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34373 every 8 connection(s)
I20260927 05:14:00.948655 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:14:00.950068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9683
I20260927 05:14:00.950276 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:14:00.950912 388 external_mini_cluster.cc:1428] Setting key 55741778d094cd7e194a73fee86ad647
I20260927 05:14:00.952641 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:00.965508 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:00.952967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.97.62:34373)
0927 05:14:00.953975 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:00.953988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:00.954039 (+ 51us) server_negotiation.cc:407] Connection header received
0927 05:14:00.955833 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:00.955860 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:00.956420 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:00.956851 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:00.957361 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.959374 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.960656 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:00.961631 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:00.961823 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:00.962870 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:00.962880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:00.962896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:00.963609 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:00.963703 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:00.963726 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:00.963839 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:00.964017 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 05:14:00.964318 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":306,"threads_started":1}
I20260927 05:14:00.966643 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b: Bootstrap starting.
I20260927 05:14:00.969511 9749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:00.970414 9749 log.cc:824] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:00.972849 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b: No bootstrap required, opened a new log
I20260927 05:14:00.980134 9749 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260927 05:14:00.980495 9749 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:00.980571 9749 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad973af6b3ca4276a94c970628c5433b, State: Initialized, Role: FOLLOWER
I20260927 05:14:00.981012 9749 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260927 05:14:00.981194 9749 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:14:00.981283 9749 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:14:00.981397 9749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:00.983361 9749 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260927 05:14:00.983758 9749 leader_election.cc:302] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad973af6b3ca4276a94c970628c5433b; no voters:
I20260927 05:14:00.984238 9749 leader_election.cc:288] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:14:00.984403 9754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:00.985106 9754 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [term 1 LEADER]: Becoming Leader. State: Replica: ad973af6b3ca4276a94c970628c5433b, State: Running, Role: LEADER
I20260927 05:14:00.985659 9754 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260927 05:14:00.986224 9749 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:14:00.988571 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad973af6b3ca4276a94c970628c5433b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } } }
I20260927 05:14:00.988571 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad973af6b3ca4276a94c970628c5433b. Latest consensus state: current_term: 1 leader_uuid: "ad973af6b3ca4276a94c970628c5433b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad973af6b3ca4276a94c970628c5433b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } } }
I20260927 05:14:00.989008 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:00.989094 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:00.991967 9759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:14:00.996742 9759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:14:01.005638 9759 catalog_manager.cc:1347] Generated new cluster ID: dca28684137a47bcbff0ab097b08e5c8
I20260927 05:14:01.005739 9759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:14:01.018630 9759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:14:01.019919 9759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:14:01.026368 9759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad973af6b3ca4276a94c970628c5433b: Generated new TSK 0
I20260927 05:14:01.027118 9759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:14:01.037161 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--builtin_ntp_servers=127.0.97.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 05:14:01.372588 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:01.372886 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:01.373037 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:01.379978 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:01.380095 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:01.380252 9773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:14:01.389014 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:01.390177 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:01.391947 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:01.404464 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:01.404445 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:01.404493 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:01.405241 9773 server_base.cc:1034] running on GCE node
I20260927 05:14:01.405807 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:01.406692 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:01.407909 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790486041407869 us; error 41 us; skew 500 ppm
I20260927 05:14:01.408258 9773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:01.410396 9773 webserver.cc:460] Webserver started at http://127.0.97.1:46529/ using document root <none> and password file <none>
I20260927 05:14:01.411083 9773 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:01.411209 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:01.411527 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:01.414120 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "db2af0a287f94b749e40b5c418f65d52"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "471153e64e2838365b28666c41c49800"
server_key_iv: "6e9f19e45ebf4f22e38bfc8fcd581ac1"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.414878 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "db2af0a287f94b749e40b5c418f65d52"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "471153e64e2838365b28666c41c49800"
server_key_iv: "6e9f19e45ebf4f22e38bfc8fcd581ac1"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.419407 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:14:01.422402 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.423629 9773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:14:01.423799 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "db2af0a287f94b749e40b5c418f65d52"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "471153e64e2838365b28666c41c49800"
server_key_iv: "6e9f19e45ebf4f22e38bfc8fcd581ac1"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.424009 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:01.436478 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:01.437201 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:01.437477 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:01.438365 9773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:01.439749 9773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:01.439877 9773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.440012 9773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:01.440100 9773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.471055 9773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41633
I20260927 05:14:01.471136 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41633 every 8 connection(s)
I20260927 05:14:01.472312 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:14:01.477746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9773
I20260927 05:14:01.477988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:14:01.478667 388 external_mini_cluster.cc:1428] Setting key 6d3b79cc6402121c71024c466beeb22a
I20260927 05:14:01.481240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--builtin_ntp_servers=127.0.97.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:14:01.485002 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:01.474671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34949 (local address 127.0.97.62:34373)
0927 05:14:01.475104 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:01.475108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:01.476130 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 05:14:01.477855 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:01.477860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:01.477950 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:01.478069 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:01.479256 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.480115 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.481799 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.482562 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.482705 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:01.484049 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:01.484057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:01.484065 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:01.484087 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:01.484150 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:01.484169 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:01.484263 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:01.484473 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:14:01.484576 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":218,"threads_started":1}
I20260927 05:14:01.485708 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:01.475135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34373 (local address 127.0.97.1:34949)
0927 05:14:01.475977 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:01.476015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:01.477603 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:01.478302 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:01.478309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:01.478447 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:01.479094 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:01.479112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.480272 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.480278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:01.481660 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:01.481672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.482764 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.482769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:01.482970 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:01.483750 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:01.483753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:01.483792 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:01.483870 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:01.484267 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:01.484274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:01.484280 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:14:01.484417 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 05:14:01.484685 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":268,"threads_started":1}
I20260927 05:14:01.487234 9900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34373
I20260927 05:14:01.487617 9900 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:01.488320 9900 heartbeater.cc:511] Master 127.0.97.62:34373 requested a full tablet report, sending...
I20260927 05:14:01.490423 9714 ts_manager.cc:194] Registered new tserver with Master: db2af0a287f94b749e40b5c418f65d52 (127.0.97.1:41633)
I20260927 05:14:01.492525 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34949
W20260927 05:14:01.799595 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:01.799853 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:01.799932 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:01.807049 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:01.807153 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:01.807256 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:14:01.816566 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:01.817701 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:01.819432 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:01.832113 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:01.832167 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:01.832191 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:01.833029 9904 server_base.cc:1034] running on GCE node
I20260927 05:14:01.833676 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:01.834558 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:01.835781 9904 hybrid_clock.cc:648] HybridClock initialized: now 1790486041835746 us; error 53 us; skew 500 ppm
I20260927 05:14:01.836097 9904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:01.838147 9904 webserver.cc:460] Webserver started at http://127.0.97.2:39523/ using document root <none> and password file <none>
I20260927 05:14:01.838796 9904 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:01.838949 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:01.839282 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:01.841941 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "43e7b43d48e64951a0cd14c2f5a42c51"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "fb190e6ea7bf0c5ff4b863051b0418e6"
server_key_iv: "141f3ee57641106c608a2d19db999a13"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.842700 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "43e7b43d48e64951a0cd14c2f5a42c51"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "fb190e6ea7bf0c5ff4b863051b0418e6"
server_key_iv: "141f3ee57641106c608a2d19db999a13"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.847541 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:14:01.850636 9918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.851862 9904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:14:01.852033 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "43e7b43d48e64951a0cd14c2f5a42c51"
format_stamp: "Formatted at 2026-09-27 05:14:01 on dist-test-slave-1t6p"
server_key: "fb190e6ea7bf0c5ff4b863051b0418e6"
server_key_iv: "141f3ee57641106c608a2d19db999a13"
server_key_version: "encryptionkey@0"
I20260927 05:14:01.852228 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:01.865088 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:01.865861 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:01.866135 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:01.867094 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:01.868460 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:01.868584 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.868721 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:01.868808 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:01.900921 9904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35777
I20260927 05:14:01.900976 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35777 every 8 connection(s)
I20260927 05:14:01.902263 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:14:01.904635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 9904
I20260927 05:14:01.904805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:14:01.905303 388 external_mini_cluster.cc:1428] Setting key d13324448d952675de92492f312e32cc
I20260927 05:14:01.908113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--builtin_ntp_servers=127.0.97.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:14:01.915777 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:01.904843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55551 (local address 127.0.97.62:34373)
0927 05:14:01.905074 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:01.905080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:01.906782 (+ 1702us) server_negotiation.cc:407] Connection header received
0927 05:14:01.908894 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:01.908902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:01.909001 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:01.909134 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:01.910211 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.911366 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.912877 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.913282 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.913474 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:01.914804 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:01.914812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:01.914820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:01.914843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:01.914886 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:01.914904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:01.915021 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:01.915305 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 05:14:01.915419 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:14:01.916648 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:01.905462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34373 (local address 127.0.97.2:55551)
0927 05:14:01.906609 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:01.906664 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:01.908578 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:01.909366 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:01.909379 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:01.909468 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:01.910044 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:01.910060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.911503 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.911507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:01.912717 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:01.912728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:01.913561 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:01.913568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:01.913789 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:01.914492 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:01.914495 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:01.914533 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:01.914603 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:01.915079 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:01.915089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:01.915098 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 05:14:01.915261 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 05:14:01.915543 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":401,"threads_started":1}
I20260927 05:14:01.918037 10031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34373
I20260927 05:14:01.918336 10031 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:01.919098 10031 heartbeater.cc:511] Master 127.0.97.62:34373 requested a full tablet report, sending...
I20260927 05:14:01.920516 9714 ts_manager.cc:194] Registered new tserver with Master: 43e7b43d48e64951a0cd14c2f5a42c51 (127.0.97.2:35777)
I20260927 05:14:01.921298 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55551
W20260927 05:14:02.226190 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:02.226457 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:02.226590 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:02.234081 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:02.234181 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:02.234350 10034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:14:02.243345 10034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:02.244597 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:02.246285 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:02.258908 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:02.258956 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:02.259233 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:02.259724 10034 server_base.cc:1034] running on GCE node
I20260927 05:14:02.260257 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:02.261046 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:02.262231 10034 hybrid_clock.cc:648] HybridClock initialized: now 1790486042262172 us; error 51 us; skew 500 ppm
I20260927 05:14:02.262548 10034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:02.268829 10034 webserver.cc:460] Webserver started at http://127.0.97.3:34949/ using document root <none> and password file <none>
I20260927 05:14:02.269479 10034 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:02.269590 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:02.269912 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:02.272584 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "375f93b641f64dc897e699c95053e946"
format_stamp: "Formatted at 2026-09-27 05:14:02 on dist-test-slave-1t6p"
server_key: "dae6f5c10304377eee1cfc6badc53163"
server_key_iv: "bff85c038eb6a5576a04d1ade7778ce0"
server_key_version: "encryptionkey@0"
I20260927 05:14:02.273324 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "375f93b641f64dc897e699c95053e946"
format_stamp: "Formatted at 2026-09-27 05:14:02 on dist-test-slave-1t6p"
server_key: "dae6f5c10304377eee1cfc6badc53163"
server_key_iv: "bff85c038eb6a5576a04d1ade7778ce0"
server_key_version: "encryptionkey@0"
I20260927 05:14:02.278067 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 05:14:02.281186 10048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:02.282428 10034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:14:02.282594 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "375f93b641f64dc897e699c95053e946"
format_stamp: "Formatted at 2026-09-27 05:14:02 on dist-test-slave-1t6p"
server_key: "dae6f5c10304377eee1cfc6badc53163"
server_key_iv: "bff85c038eb6a5576a04d1ade7778ce0"
server_key_version: "encryptionkey@0"
I20260927 05:14:02.282787 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:02.298352 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:02.299178 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:02.299438 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:02.300274 10034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:02.301594 10034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:02.301703 10034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:02.301837 10034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:02.301937 10034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:02.334434 10034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39167
I20260927 05:14:02.334483 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39167 every 8 connection(s)
I20260927 05:14:02.335743 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:14:02.340538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10034
I20260927 05:14:02.340720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:14:02.341233 388 external_mini_cluster.cc:1428] Setting key f0ccdfeb292e1d54c436d64187ef1b49
I20260927 05:14:02.348240 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:02.338321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36529 (local address 127.0.97.62:34373)
0927 05:14:02.338496 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:02.338501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:02.339883 (+ 1382us) server_negotiation.cc:407] Connection header received
0927 05:14:02.342216 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:02.342219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:02.342284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:02.342410 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:02.343662 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.344588 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.346000 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.346387 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.346472 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:02.347642 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:02.347646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:02.347650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:02.347666 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:02.347691 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:02.347702 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:02.347770 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:02.347956 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 05:14:02.348022 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:14:02.349149 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:02.338828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34373 (local address 127.0.97.3:36529)
0927 05:14:02.339743 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:02.339777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:02.341950 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:02.342649 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:02.342657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:02.342737 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:02.343508 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:02.343531 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.344714 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.344718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:02.345880 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:02.345891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.346552 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.346557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:02.346742 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:02.347385 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:02.347388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:02.347421 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:02.347485 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:02.347780 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:02.347787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:02.347792 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:14:02.347913 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 05:14:02.348165 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":360,"threads_started":1}
I20260927 05:14:02.350366 10161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34373
I20260927 05:14:02.350600 10161 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:02.351303 10161 heartbeater.cc:511] Master 127.0.97.62:34373 requested a full tablet report, sending...
I20260927 05:14:02.352456 9714 ts_manager.cc:194] Registered new tserver with Master: 375f93b641f64dc897e699c95053e946 (127.0.97.3:39167)
I20260927 05:14:02.353149 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36529
I20260927 05:14:02.356375 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:14:02.362133 9902 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:14:02.360697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.0.97.62:34373)
0927 05:14:02.360881 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:02.360889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:02.361349 (+ 460us) server_negotiation.cc:407] Connection header received
0927 05:14:02.361452 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:02.361455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:02.361506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:02.361579 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:02.361920 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37286: BlockingRecv error: recv got EOF from 127.0.0.1:37286 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:14:02.362737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9773
I20260927 05:14:02.369791 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:02.363656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.97.1:41633)
0927 05:14:02.364222 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:02.364229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:02.364252 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:14:02.364306 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:02.364311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:02.364470 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:02.364582 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:02.364878 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.365842 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.366970 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.367523 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.367617 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:02.368602 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:02.368608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:02.368618 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:02.369120 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:02.369157 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:02.369178 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:02.369261 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:02.369415 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 05:14:02.369520 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":318,"threads_started":1}
I20260927 05:14:02.370828 9895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:02.450346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9773
I20260927 05:14:02.466661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9904
I20260927 05:14:02.474395 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:02.467549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.97.2:35777)
0927 05:14:02.468136 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:02.468143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:02.468169 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:14:02.468233 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:02.468239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:02.468448 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:02.468580 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:02.469028 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.470001 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.471315 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.471884 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.471986 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:02.472984 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:02.472992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:02.473005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:02.473596 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:02.473639 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:02.473662 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:02.473754 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:02.473893 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:14:02.474047 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
I20260927 05:14:02.475662 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:02.554538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9904
I20260927 05:14:02.570729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10034
I20260927 05:14:02.578274 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:02.571764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.0.97.3:39167)
0927 05:14:02.572400 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:02.572407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:02.572448 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:14:02.572510 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:02.572517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:02.572726 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:02.572855 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:02.573211 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.574231 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.575450 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:02.575980 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:02.576077 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:02.577038 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:02.577045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:02.577057 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:02.577614 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:02.577661 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:02.577679 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:02.577764 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:02.577873 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 05:14:02.577981 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":360,"threads_started":1}
I20260927 05:14:02.579481 10156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:02.659955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10034
I20260927 05:14:02.675832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 9683
I20260927 05:14:02.676906 9744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:02.775943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 9683
2026-09-27T05:14:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2302 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T05:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:14:02Z Disabled control of system clock
I20260927 05:14:02.842934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42829
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:14:03.151990 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:03.152273 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:03.152361 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:03.160492 10180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:14:03.160593 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:03.160656 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:03.160733 10180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:14:03.160810 10180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:14:03.170760 10180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:03.172080 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:03.173786 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:03.186365 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:03.186316 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:03.186415 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:03.188372 10180 server_base.cc:1034] running on GCE node
I20260927 05:14:03.189082 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:03.190367 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:03.191551 10180 hybrid_clock.cc:648] HybridClock initialized: now 1790486043191496 us; error 46 us; skew 500 ppm
I20260927 05:14:03.191869 10180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:03.193938 10180 webserver.cc:460] Webserver started at http://127.0.97.62:37201/ using document root <none> and password file <none>
I20260927 05:14:03.194571 10180 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:03.194687 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:03.195063 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:03.197719 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "1e13defd1dbd4044b0013d55253ad903"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "a54c931be4569090807fc21b6064fca9"
server_key_iv: "43a5b932161625ddf50b8f0a798bf4dc"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.198475 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "1e13defd1dbd4044b0013d55253ad903"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "a54c931be4569090807fc21b6064fca9"
server_key_iv: "43a5b932161625ddf50b8f0a798bf4dc"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.203262 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 05:14:03.206351 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:03.207677 10180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:14:03.207845 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "1e13defd1dbd4044b0013d55253ad903"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "a54c931be4569090807fc21b6064fca9"
server_key_iv: "43a5b932161625ddf50b8f0a798bf4dc"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.208065 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:03.213510 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:03.214224 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:03.214521 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:03.232291 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42829
I20260927 05:14:03.232323 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42829 every 8 connection(s)
I20260927 05:14:03.233548 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:14:03.237048 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:03.240154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10180
I20260927 05:14:03.240352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:14:03.240973 388 external_mini_cluster.cc:1428] Setting key 8f66b931ce7cbabaaa55e8314a4ed683
I20260927 05:14:03.248596 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:03.242816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.0.97.62:42829)
0927 05:14:03.243802 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:03.243813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:03.243851 (+ 38us) server_negotiation.cc:407] Connection header received
0927 05:14:03.245325 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:03.245346 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:03.245743 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:03.246075 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:03.247129 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:03.247138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:03.247152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:03.247166 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:03.247220 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:03.247235 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:03.247398 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:03.247532 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 05:14:03.247791 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":413,"threads_started":1}
I20260927 05:14:03.249205 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903: Bootstrap starting.
I20260927 05:14:03.252192 10246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:03.253058 10246 log.cc:824] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:03.255352 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903: No bootstrap required, opened a new log
I20260927 05:14:03.262209 10246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260927 05:14:03.262574 10246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:03.262667 10246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e13defd1dbd4044b0013d55253ad903, State: Initialized, Role: FOLLOWER
I20260927 05:14:03.263166 10246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260927 05:14:03.263355 10246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:14:03.263442 10246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:14:03.263546 10246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:03.265393 10246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260927 05:14:03.265790 10246 leader_election.cc:302] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e13defd1dbd4044b0013d55253ad903; no voters:
I20260927 05:14:03.266254 10246 leader_election.cc:288] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:14:03.266404 10251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:03.267159 10251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [term 1 LEADER]: Becoming Leader. State: Replica: 1e13defd1dbd4044b0013d55253ad903, State: Running, Role: LEADER
I20260927 05:14:03.267609 10251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260927 05:14:03.268234 10246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:14:03.270181 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e13defd1dbd4044b0013d55253ad903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } } }
I20260927 05:14:03.270476 10253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e13defd1dbd4044b0013d55253ad903. Latest consensus state: current_term: 1 leader_uuid: "1e13defd1dbd4044b0013d55253ad903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13defd1dbd4044b0013d55253ad903" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } } }
I20260927 05:14:03.270852 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:03.270942 10253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:03.272179 10258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:14:03.276995 10258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:14:03.285087 10258 catalog_manager.cc:1347] Generated new cluster ID: 700976a9bcec42dd9a6f45b8e7bfea1a
I20260927 05:14:03.285189 10258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:14:03.308732 10258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:14:03.309131 10258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:14:03.328850 10258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e13defd1dbd4044b0013d55253ad903: Generated new TSK 0
I20260927 05:14:03.329622 10258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:14:03.335328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--builtin_ntp_servers=127.0.97.20:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:14:03.654539 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:03.654805 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:03.654946 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:03.664528 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:03.664688 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:03.664866 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:14:03.674111 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:03.675390 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:03.677057 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:03.690075 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:03.690047 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:03.689988 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:03.690647 10270 server_base.cc:1034] running on GCE node
I20260927 05:14:03.691303 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:03.692199 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:03.693414 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790486043693377 us; error 56 us; skew 500 ppm
I20260927 05:14:03.693718 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:03.695879 10270 webserver.cc:460] Webserver started at http://127.0.97.1:44807/ using document root <none> and password file <none>
I20260927 05:14:03.696525 10270 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:03.696652 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:03.696964 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:03.699705 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "d0949877e9074732878fb56627846e1e"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "70ee54e50a0d4c0e87b00593d3c6a8f4"
server_key_iv: "1235cec3e70add861913ca3eccbe381e"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.700479 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d0949877e9074732878fb56627846e1e"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "70ee54e50a0d4c0e87b00593d3c6a8f4"
server_key_iv: "1235cec3e70add861913ca3eccbe381e"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.705356 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 05:14:03.708720 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:03.710000 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:14:03.710168 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "d0949877e9074732878fb56627846e1e"
format_stamp: "Formatted at 2026-09-27 05:14:03 on dist-test-slave-1t6p"
server_key: "70ee54e50a0d4c0e87b00593d3c6a8f4"
server_key_iv: "1235cec3e70add861913ca3eccbe381e"
server_key_version: "encryptionkey@0"
I20260927 05:14:03.710391 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:03.716128 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:03.716873 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:03.717136 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:03.718122 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:03.719521 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:03.719638 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:03.719770 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:03.719857 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:03.752874 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36149
I20260927 05:14:03.752926 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36149 every 8 connection(s)
I20260927 05:14:03.754192 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:14:03.755751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10270
I20260927 05:14:03.755970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:14:03.756790 388 external_mini_cluster.cc:1428] Setting key 5ac47ecf20276624ad9a2fb9f9ec82de
I20260927 05:14:03.759954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--builtin_ntp_servers=127.0.97.20:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:14:03.763795 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:03.756989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43899 (local address 127.0.97.62:42829)
0927 05:14:03.757484 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:03.757490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:03.758945 (+ 1455us) server_negotiation.cc:407] Connection header received
0927 05:14:03.761512 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:03.761516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:03.761563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:03.761659 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:03.763123 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:03.763127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:03.763130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:03.763145 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:03.763185 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:03.763196 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:03.763295 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:03.763543 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 05:14:03.763597 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260927 05:14:03.765043 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:03.757662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.1:43899)
0927 05:14:03.758761 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:03.758850 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:03.761202 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:03.761924 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:03.761938 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:03.762066 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:03.762772 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:03.762778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:03.762828 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:03.762940 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:03.763302 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:03.763322 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:03.763332 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:14:03.763614 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 05:14:03.763857 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":377,"threads_started":1}
I20260927 05:14:03.766531 10397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260927 05:14:03.766844 10397 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:03.767536 10397 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
I20260927 05:14:03.769817 10211 ts_manager.cc:194] Registered new tserver with Master: d0949877e9074732878fb56627846e1e (127.0.97.1:36149)
W20260927 05:14:04.074265 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:04.074510 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:04.074594 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:04.081449 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:04.081540 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:04.081642 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:14:04.090139 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:04.091315 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:04.092940 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:04.105032 10407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:04.105062 10409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:04.105311 10406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:04.106089 10401 server_base.cc:1034] running on GCE node
I20260927 05:14:04.106593 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:04.107455 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:04.108677 10401 hybrid_clock.cc:648] HybridClock initialized: now 1790486044108625 us; error 43 us; skew 500 ppm
I20260927 05:14:04.108978 10401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:04.111243 10401 webserver.cc:460] Webserver started at http://127.0.97.2:34293/ using document root <none> and password file <none>
I20260927 05:14:04.111931 10401 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:04.112051 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:04.112360 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:04.116717 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f6d8aa094a54749a5ba444d133c53bd"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "1c87323233d91e3600e754446a0605bd"
server_key_iv: "4f370b7e392d73f1297432cdeb34d843"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.117452 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f6d8aa094a54749a5ba444d133c53bd"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "1c87323233d91e3600e754446a0605bd"
server_key_iv: "4f370b7e392d73f1297432cdeb34d843"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.122152 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 05:14:04.125304 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.126516 10401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:14:04.126679 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "3f6d8aa094a54749a5ba444d133c53bd"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "1c87323233d91e3600e754446a0605bd"
server_key_iv: "4f370b7e392d73f1297432cdeb34d843"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.126894 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:04.132128 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:04.132882 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:04.133145 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:04.133968 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:04.135279 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:04.135394 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.135531 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:04.135620 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.166919 10401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41095
I20260927 05:14:04.166971 10527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41095 every 8 connection(s)
I20260927 05:14:04.168200 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:14:04.172945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10401
I20260927 05:14:04.173105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:14:04.173561 388 external_mini_cluster.cc:1428] Setting key 36ad181819f3341c2acd7e6e402c2f97
I20260927 05:14:04.176530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--builtin_ntp_servers=127.0.97.20:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:14:04.176708 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.170777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41203 (local address 127.0.97.62:42829)
0927 05:14:04.170953 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.170957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.172316 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 05:14:04.174266 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.174273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.174362 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.174474 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.175991 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:04.175996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:04.176001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:04.176018 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:04.176053 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:04.176067 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:04.176193 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:04.176344 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 05:14:04.176407 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:14:04.177518 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.171317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.2:41203)
0927 05:14:04.172173 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:04.172207 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:04.173946 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:04.174728 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:04.174738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:04.174847 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:04.175675 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:04.175678 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:04.175715 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:04.175826 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:04.176156 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:04.176162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:04.176167 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:14:04.176374 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 05:14:04.176560 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
I20260927 05:14:04.178797 10528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260927 05:14:04.179104 10528 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:04.179734 10528 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
I20260927 05:14:04.181265 10211 ts_manager.cc:194] Registered new tserver with Master: 3f6d8aa094a54749a5ba444d133c53bd (127.0.97.2:41095)
W20260927 05:14:04.490779 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:04.491113 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:04.491214 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:04.498641 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:04.498751 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:04.498867 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:14:04.508605 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:04.509895 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:04.511780 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:04.525080 10537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:04.525080 10539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:04.525504 10536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:04.526001 10531 server_base.cc:1034] running on GCE node
I20260927 05:14:04.526566 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:04.527534 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:04.528760 10531 hybrid_clock.cc:648] HybridClock initialized: now 1790486044528714 us; error 56 us; skew 500 ppm
I20260927 05:14:04.529074 10531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:14:04.531517 10531 webserver.cc:460] Webserver started at http://127.0.97.3:44529/ using document root <none> and password file <none>
I20260927 05:14:04.532184 10531 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:04.532316 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:04.532630 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:04.535440 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "b8d94ddda5704c5d8fb2a43d7ae70a05"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "18f3b7485d46cc49ea83d0e4a23a0454"
server_key_iv: "3f53be42223749d5c4c28dfba4859673"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.536180 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b8d94ddda5704c5d8fb2a43d7ae70a05"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "18f3b7485d46cc49ea83d0e4a23a0454"
server_key_iv: "3f53be42223749d5c4c28dfba4859673"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.540964 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 05:14:04.544417 10545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.545713 10531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:14:04.545897 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "b8d94ddda5704c5d8fb2a43d7ae70a05"
format_stamp: "Formatted at 2026-09-27 05:14:04 on dist-test-slave-1t6p"
server_key: "18f3b7485d46cc49ea83d0e4a23a0454"
server_key_iv: "3f53be42223749d5c4c28dfba4859673"
server_key_version: "encryptionkey@0"
I20260927 05:14:04.546118 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:04.551687 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:04.552435 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:04.552691 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:04.553572 10531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:04.554852 10531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:04.554970 10531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.555184 10531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:04.555280 10531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:04.588347 10531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40101
I20260927 05:14:04.588390 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40101 every 8 connection(s)
I20260927 05:14:04.589645 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:14:04.598688 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.592183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34349 (local address 127.0.97.62:42829)
0927 05:14:04.592353 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.592357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.594616 (+ 2259us) server_negotiation.cc:407] Connection header received
0927 05:14:04.596531 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.596536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.596610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.596720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.597958 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:04.597964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:04.597969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:04.597987 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:04.598025 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:04.598036 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:04.598163 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:04.598351 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 05:14:04.598426 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:14:04.599196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10531
I20260927 05:14:04.599355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:14:04.599704 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.592686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.3:34349)
0927 05:14:04.594314 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:04.594397 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:04.596268 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:04.596955 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:04.596964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:04.597051 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:04.597670 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:14:04.597673 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:04.597713 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:14:04.597798 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:04.598127 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:04.598136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:04.598147 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:14:04.598417 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 05:14:04.598645 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":413,"threads_started":1}
I20260927 05:14:04.599848 388 external_mini_cluster.cc:1428] Setting key 32d99d62776ce663c0a9face88102e7e
I20260927 05:14:04.601207 10658 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260927 05:14:04.601542 10658 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:04.602211 10658 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
I20260927 05:14:04.603618 10211 ts_manager.cc:194] Registered new tserver with Master: b8d94ddda5704c5d8fb2a43d7ae70a05 (127.0.97.3:40101)
I20260927 05:14:04.614949 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:14:04.620636 10399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:14:04.619177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.0.97.62:42829)
0927 05:14:04.619388 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.619395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.619877 (+ 482us) server_negotiation.cc:407] Connection header received
0927 05:14:04.619995 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.619999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.620043 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.620118 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.620409 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47648: BlockingRecv error: recv got EOF from 127.0.0.1:47648 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:14:04.621054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10270
I20260927 05:14:04.624380 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.622018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.0.97.1:36149)
0927 05:14:04.622520 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.622526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.622550 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:14:04.622601 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.622607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.622791 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.622911 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.623812 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:04.623818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:04.623829 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:04.623846 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:04.623895 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:04.623908 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:04.624008 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:04.624097 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 05:14:04.624167 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":252,"threads_started":1}
I20260927 05:14:04.625551 10392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:04.704546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10270
I20260927 05:14:04.720919 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10401
I20260927 05:14:04.724299 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.721771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.0.97.2:41095)
0927 05:14:04.722260 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.722266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.722292 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:14:04.722363 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.722369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.722602 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.722728 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.723591 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:04.723599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:04.723610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:04.723622 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:04.723664 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:04.723683 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:04.723817 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:04.723931 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 05:14:04.724050 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260927 05:14:04.725499 10523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:04.804947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10401
I20260927 05:14:04.820600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10531
I20260927 05:14:04.824290 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:04.821551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.0.97.3:40101)
0927 05:14:04.822169 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:04.822176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:04.822204 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:14:04.822281 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:04.822287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:04.822512 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:04.822633 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:04.823590 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:04.823597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:04.823608 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:14:04.823624 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:04.823660 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:14:04.823672 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:04.823800 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:04.823851 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 05:14:04.823962 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":334,"threads_started":1}
I20260927 05:14:04.825439 10653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:04.904249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10531
I20260927 05:14:04.919589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10180
I20260927 05:14:04.920723 10241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:05.019099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10180
2026-09-27T05:14:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2242 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:14:05 dist-test-slave-1t6p krb5kdc[10676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:14:05 dist-test-slave-1t6p krb5kdc[10676](info): set up 2 sockets
Sep 27 05:14:05 dist-test-slave-1t6p krb5kdc[10676](info): commencing operation
krb5kdc: starting...
W20260927 05:14:06.131783 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:14:06 dist-test-slave-1t6p krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:14:06.334713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43671
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:14:06.637962 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:06.638258 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:06.638440 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:06.645589 10689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:14:06.645716 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:06.645864 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:06.645952 10689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:14:06.646016 10689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:14:06.654865 10689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:06.656075 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:06.657315 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:06.668661 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:06.668656 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:06.668967 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:06.669385 10689 server_base.cc:1034] running on GCE node
I20260927 05:14:06.670056 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:06.671432 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:06.672659 10689 hybrid_clock.cc:648] HybridClock initialized: now 1790486046672606 us; error 41 us; skew 500 ppm
I20260927 05:14:06.672993 10689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:06 dist-test-slave-1t6p krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:06.675877 10689 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:14:06.677145 10689 webserver.cc:460] Webserver started at http://127.0.97.62:42243/ using document root <none> and password file <none>
I20260927 05:14:06.677723 10689 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:06.677816 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:06.678045 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:06.680755 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "c5aa2e18a51c454abc6502911cedb080"
format_stamp: "Formatted at 2026-09-27 05:14:06 on dist-test-slave-1t6p"
server_key: "29eb0159dd63c473e6fa4f2bc3bed136"
server_key_iv: "ab7f8bb308c52da3bc72e6eec2f5b2c6"
server_key_version: "encryptionkey@0"
I20260927 05:14:06.681416 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5aa2e18a51c454abc6502911cedb080"
format_stamp: "Formatted at 2026-09-27 05:14:06 on dist-test-slave-1t6p"
server_key: "29eb0159dd63c473e6fa4f2bc3bed136"
server_key_iv: "ab7f8bb308c52da3bc72e6eec2f5b2c6"
server_key_version: "encryptionkey@0"
I20260927 05:14:06.686180 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:14:06.689251 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:06.690477 10689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:14:06.690641 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "c5aa2e18a51c454abc6502911cedb080"
format_stamp: "Formatted at 2026-09-27 05:14:06 on dist-test-slave-1t6p"
server_key: "29eb0159dd63c473e6fa4f2bc3bed136"
server_key_iv: "ab7f8bb308c52da3bc72e6eec2f5b2c6"
server_key_version: "encryptionkey@0"
I20260927 05:14:06.690814 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:06.705140 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:06.708684 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:06.708961 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:06.726027 10689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43671
I20260927 05:14:06.726058 10755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43671 every 8 connection(s)
I20260927 05:14:06.727303 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:14:06.730515 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:06.732612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10689
I20260927 05:14:06.732798 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:14:06.733428 388 external_mini_cluster.cc:1428] Setting key 03c12b73f749ee59ccd06501e994fb1c
I20260927 05:14:06.742787 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: Bootstrap starting.
I20260927 05:14:06.746054 10756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:06.746984 10756 log.cc:824] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:14:06 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:06.749841 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: No bootstrap required, opened a new log
I20260927 05:14:06.756907 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:06.735312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.97.62:43671)
0927 05:14:06.736074 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:06.736089 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:06.736136 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:14:06.738309 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:06.738335 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:06.738767 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:06.739236 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:06.740433 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:06.742307 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:06.743712 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:06.744780 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:06.744986 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:06.749018 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:06.749030 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:06.749050 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:06.749093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:06.752873 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:06.753498 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:06.753506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:06.753514 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:06.753718 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:06.754032 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:06.754037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:06.754039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:06.754654 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:06.754949 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:06.755256 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 05:14:06.755666 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":308,"threads_started":1}
I20260927 05:14:06.760203 10756 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } }
I20260927 05:14:06.760499 10756 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:06.760566 10756 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5aa2e18a51c454abc6502911cedb080, State: Initialized, Role: FOLLOWER
I20260927 05:14:06.761111 10756 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } }
I20260927 05:14:06.761389 10756 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:14:06.761550 10756 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:14:06.761718 10756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:06.763834 10756 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } }
I20260927 05:14:06.764276 10756 leader_election.cc:302] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5aa2e18a51c454abc6502911cedb080; no voters:
I20260927 05:14:06.764804 10756 leader_election.cc:288] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:14:06.764941 10761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:06.765695 10761 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [term 1 LEADER]: Becoming Leader. State: Replica: c5aa2e18a51c454abc6502911cedb080, State: Running, Role: LEADER
I20260927 05:14:06.766281 10761 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } }
I20260927 05:14:06.766957 10756 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:14:06.768886 10763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5aa2e18a51c454abc6502911cedb080. Latest consensus state: current_term: 1 leader_uuid: "c5aa2e18a51c454abc6502911cedb080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } } }
I20260927 05:14:06.769225 10763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:06.770232 10768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:14:06.769932 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5aa2e18a51c454abc6502911cedb080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5aa2e18a51c454abc6502911cedb080" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43671 } } }
I20260927 05:14:06.770560 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:06.779083 10768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:14:06.788942 10768 catalog_manager.cc:1347] Generated new cluster ID: f39fef7fc8ac41cd860104c5d41772f4
I20260927 05:14:06.789098 10768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:14:06.798329 10768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:14:06.800034 10768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:14:06.820271 10768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: Generated new TSK 0
I20260927 05:14:06.821171 10768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:14:06.880713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:14:07.198086 10782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:07.198490 10782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:07.198650 10782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:07.205926 10782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:07.206017 10782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:07.206207 10782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:14:07.215922 10782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:07.217157 10782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:07.218456 10782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:07.231216 10787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:07.231282 10788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:07.231309 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:07.232311 10782 server_base.cc:1034] running on GCE node
I20260927 05:14:07.232889 10782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:07.233747 10782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:07.234966 10782 hybrid_clock.cc:648] HybridClock initialized: now 1790486047234918 us; error 61 us; skew 500 ppm
I20260927 05:14:07.235299 10782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:07 dist-test-slave-1t6p krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:07.238250 10782 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:14:07.239641 10782 webserver.cc:460] Webserver started at http://127.0.97.1:39919/ using document root <none> and password file <none>
I20260927 05:14:07.240243 10782 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:07.240335 10782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:07.240561 10782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:07.243221 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "57dedf2e42a24572a577127fa8451c8c"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "6740d9efd3ce6d1fd2a9b7e4f29b7f0a"
server_key_iv: "84f74ea3d89e52607f3abdba0c25badd"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.243868 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "57dedf2e42a24572a577127fa8451c8c"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "6740d9efd3ce6d1fd2a9b7e4f29b7f0a"
server_key_iv: "84f74ea3d89e52607f3abdba0c25badd"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.248435 10782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:14:07.251533 10797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.252728 10782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 05:14:07.252849 10782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "57dedf2e42a24572a577127fa8451c8c"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "6740d9efd3ce6d1fd2a9b7e4f29b7f0a"
server_key_iv: "84f74ea3d89e52607f3abdba0c25badd"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.253010 10782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:07.277546 10782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:07.281100 10782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:07.281311 10782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:07.282085 10782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:07.283336 10782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:07.283409 10782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.283490 10782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:07.283574 10782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.315411 10782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37577
I20260927 05:14:07.315500 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37577 every 8 connection(s)
I20260927 05:14:07.316602 10782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:14:07.322654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10782
I20260927 05:14:07.322878 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:14:07.323542 388 external_mini_cluster.cc:1428] Setting key 4d6af3c5f9e44735f8839dced8b15520
Sep 27 05:14:07 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:07.336122 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:07.318923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56603 (local address 127.0.97.62:43671)
0927 05:14:07.319336 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:07.319341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:07.320465 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 05:14:07.322148 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:07.322154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:07.322241 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:07.322364 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:07.323988 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.324836 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.326371 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.326747 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.326878 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:07.330325 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:07.330334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:07.330342 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:07.330384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:07.333102 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:07.333989 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:07.333993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:07.333996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:07.334096 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:07.334532 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:07.334535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:07.334537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:07.334759 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:07.334857 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:07.335736 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 05:14:07.335878 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":252,"threads_started":1}
I20260927 05:14:07.337560 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:07.319571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43671 (local address 127.0.97.1:56603)
0927 05:14:07.320301 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:07.320342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:07.321899 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:07.322605 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:07.322617 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:07.323253 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:07.323822 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:07.323838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.324989 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.324994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:07.326220 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:07.326231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.326958 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.326967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:07.327207 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:07.327849 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:07.327869 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:07.330126 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:07.333234 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:07.333245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:07.333262 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:07.333866 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:07.334192 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:07.334200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:07.334206 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:07.334426 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:07.334885 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:07.334895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:07.335364 (+ 469us) client_negotiation.cc:769] Sending connection context
0927 05:14:07.335642 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 05:14:07.336180 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":281,"threads_started":1}
I20260927 05:14:07.339025 10910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43671
I20260927 05:14:07.339354 10910 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:07.340147 10910 heartbeater.cc:511] Master 127.0.97.62:43671 requested a full tablet report, sending...
I20260927 05:14:07.342146 10721 ts_manager.cc:194] Registered new tserver with Master: 57dedf2e42a24572a577127fa8451c8c (127.0.97.1:37577)
I20260927 05:14:07.343581 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56603
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:14:07.374302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:14:07.692482 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:07.692770 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:07.692869 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:07.700410 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:07.700516 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:07.700624 10916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:14:07.709592 10916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:07.710683 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:07.712134 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:07.724324 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:07.724366 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:07.724577 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:07.725008 10916 server_base.cc:1034] running on GCE node
I20260927 05:14:07.725605 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:07.726460 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:07.727669 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790486047727618 us; error 44 us; skew 500 ppm
I20260927 05:14:07.727960 10916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:07 dist-test-slave-1t6p krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:07.730836 10916 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:14:07.733263 10916 webserver.cc:460] Webserver started at http://127.0.97.2:34509/ using document root <none> and password file <none>
I20260927 05:14:07.733888 10916 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:07.734014 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:07.734313 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:07.736995 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "c7d038c36be54a56a338e321b0c0f4ae"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "67ec17b8ef895a2e737836e7c1540e25"
server_key_iv: "b7d6f90de279a2cfc33ff06135c0e14d"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.737704 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c7d038c36be54a56a338e321b0c0f4ae"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "67ec17b8ef895a2e737836e7c1540e25"
server_key_iv: "b7d6f90de279a2cfc33ff06135c0e14d"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.742463 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 05:14:07.745596 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.746829 10916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:14:07.746984 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "c7d038c36be54a56a338e321b0c0f4ae"
format_stamp: "Formatted at 2026-09-27 05:14:07 on dist-test-slave-1t6p"
server_key: "67ec17b8ef895a2e737836e7c1540e25"
server_key_iv: "b7d6f90de279a2cfc33ff06135c0e14d"
server_key_version: "encryptionkey@0"
I20260927 05:14:07.747246 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:07.768348 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:07.771908 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:07.772179 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:07.773069 10916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:07.774282 10916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:07.774354 10916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.774494 10916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:07.774554 10916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:07.806340 10916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40333
I20260927 05:14:07.806420 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40333 every 8 connection(s)
I20260927 05:14:07.807610 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:14:07.813445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 10916
I20260927 05:14:07.813627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:14:07.814037 388 external_mini_cluster.cc:1428] Setting key 4dc63d92c5a3700459521ccdeb7e240f
Sep 27 05:14:07 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:07.825969 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:07.809952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56241 (local address 127.0.97.62:43671)
0927 05:14:07.810138 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:07.810145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:07.811212 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 05:14:07.812877 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:07.812881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:07.812942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:07.813027 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:07.814627 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.815452 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.816899 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.817293 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.817432 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:07.820646 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:07.820651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:07.820654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:07.820683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:07.823173 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:07.823831 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:07.823838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:07.823841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:07.823974 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:07.824405 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:07.824410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:07.824414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:07.824656 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:07.824797 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:07.825395 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 05:14:07.825591 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:14:07.827136 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:07.810385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43671 (local address 127.0.97.2:56241)
0927 05:14:07.811076 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:07.811112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:07.812643 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:07.813240 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:07.813247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:07.813828 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:07.814387 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:07.814402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.815593 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.815599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:07.816757 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:07.816768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:07.817499 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:07.817505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:07.817713 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:07.818347 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:07.818368 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:07.820424 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:07.823311 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:07.823319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:07.823334 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:07.823715 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:07.824091 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:07.824114 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:07.824118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:07.824289 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:07.824773 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:07.824780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:07.825071 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 05:14:07.825315 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 05:14:07.825823 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":247,"threads_started":1}
I20260927 05:14:07.828534 11044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43671
I20260927 05:14:07.828819 11044 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:07.829458 11044 heartbeater.cc:511] Master 127.0.97.62:43671 requested a full tablet report, sending...
I20260927 05:14:07.830869 10721 ts_manager.cc:194] Registered new tserver with Master: c7d038c36be54a56a338e321b0c0f4ae (127.0.97.2:40333)
I20260927 05:14:07.831687 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56241
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:14:07.862325 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:14:08.174441 11049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:08.174831 11049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:08.174989 11049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:08.181953 11049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:08.182041 11049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:08.182235 11049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:14:08.191567 11049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:08.192724 11049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:08.193988 11049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:08.207394 11057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:08.207434 11055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:08.207638 11054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:08.208318 11049 server_base.cc:1034] running on GCE node
I20260927 05:14:08.208911 11049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:08.209841 11049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:08.211099 11049 hybrid_clock.cc:648] HybridClock initialized: now 1790486048211046 us; error 55 us; skew 500 ppm
I20260927 05:14:08.211411 11049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:08 dist-test-slave-1t6p krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:08.214620 11049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:14:08.215909 11049 webserver.cc:460] Webserver started at http://127.0.97.3:37421/ using document root <none> and password file <none>
I20260927 05:14:08.216538 11049 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:08.216631 11049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:08.216861 11049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:14:08.219625 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "144bfcdc4d0447e599dc913b774ad103"
format_stamp: "Formatted at 2026-09-27 05:14:08 on dist-test-slave-1t6p"
server_key: "c65c08ac5861a00b14b84b798f4787dc"
server_key_iv: "f8d63515fc6c5b54b953c6935abacc43"
server_key_version: "encryptionkey@0"
I20260927 05:14:08.220294 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "144bfcdc4d0447e599dc913b774ad103"
format_stamp: "Formatted at 2026-09-27 05:14:08 on dist-test-slave-1t6p"
server_key: "c65c08ac5861a00b14b84b798f4787dc"
server_key_iv: "f8d63515fc6c5b54b953c6935abacc43"
server_key_version: "encryptionkey@0"
I20260927 05:14:08.225116 11049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:14:08.228513 11064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:08.229993 11049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:14:08.230147 11049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "144bfcdc4d0447e599dc913b774ad103"
format_stamp: "Formatted at 2026-09-27 05:14:08 on dist-test-slave-1t6p"
server_key: "c65c08ac5861a00b14b84b798f4787dc"
server_key_iv: "f8d63515fc6c5b54b953c6935abacc43"
server_key_version: "encryptionkey@0"
I20260927 05:14:08.230341 11049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:08.244414 11049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:08.247956 11049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:08.248212 11049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:08.249029 11049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:08.250223 11049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:08.250315 11049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:08.250466 11049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:08.250538 11049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:08.282229 11049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38977
I20260927 05:14:08.282274 11176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38977 every 8 connection(s)
I20260927 05:14:08.283478 11049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:14:08.290617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 11049
I20260927 05:14:08.290787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:14:08.291296 388 external_mini_cluster.cc:1428] Setting key ec762286724b8a213e926153a56dadf6
Sep 27 05:14:08 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:08.302134 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.285867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52179 (local address 127.0.97.62:43671)
0927 05:14:08.286029 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.286033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.287233 (+ 1200us) server_negotiation.cc:407] Connection header received
0927 05:14:08.288994 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.289000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.289081 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.289196 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:08.290580 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.291526 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.293298 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.293820 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.293953 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.297175 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:08.297179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:08.297183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:08.297210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:08.299795 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.300444 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.300447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.300449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.300561 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.300895 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.300898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.300900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.301118 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:08.301218 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.301770 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.301908 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:14:08.302989 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.286336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43671 (local address 127.0.97.3:52179)
0927 05:14:08.287095 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.287131 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.288727 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.289419 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.289427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.289899 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:08.290424 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.290439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.291672 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.291677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.293163 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.293175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.294061 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.294069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.294307 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.294958 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:08.294978 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:08.296986 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:08.299926 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:08.299932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:08.299942 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:08.300333 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:08.300648 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:08.300651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:08.300653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:08.300811 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:08.301212 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:08.301217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:08.301483 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.301688 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.302002 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260927 05:14:08.304229 11177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43671
I20260927 05:14:08.304528 11177 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:08.305351 11177 heartbeater.cc:511] Master 127.0.97.62:43671 requested a full tablet report, sending...
I20260927 05:14:08.306696 10721 ts_manager.cc:194] Registered new tserver with Master: 144bfcdc4d0447e599dc913b774ad103 (127.0.97.3:38977)
I20260927 05:14:08.307471 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52179
I20260927 05:14:08.317951 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:14:08.333979 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.322113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.97.62:43671)
0927 05:14:08.322287 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.322296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.322730 (+ 434us) server_negotiation.cc:407] Connection header received
0927 05:14:08.322845 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.322849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.322901 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.322979 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:08.324318 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.324967 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.326087 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.326477 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.326561 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.328614 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:08.328618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:08.328621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:08.328647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:08.331188 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.331814 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.331822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.331825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.331955 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.332430 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.332436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.332443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.332923 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:08.333215 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.333493 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.333649 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:14:08.337788 10721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:14:08.344575 10721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:14:08.346791 10910 heartbeater.cc:503] Master 127.0.97.62:43671 was elected leader, sending a full tablet report...
I20260927 05:14:08.376565 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.355866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38977 (local address 127.0.0.1:36848)
0927 05:14:08.356595 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.356637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.356838 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.369147 (+ 12309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.369151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.369170 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:08.369569 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.369583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.371540 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.371550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.373708 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.373719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.375646 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.375653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.375839 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.375886 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.375988 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.376182 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":617,"threads_started":1}
I20260927 05:14:08.377538 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.356137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.97.3:38977)
0927 05:14:08.367183 (+ 11046us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.367196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.367467 (+ 271us) server_negotiation.cc:407] Connection header received
0927 05:14:08.367585 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.367597 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.367901 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.368148 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:08.369750 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.371387 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.373906 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.375399 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.376800 (+ 1401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.377040 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.377113 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.377267 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10810,"thread_start_us":11090,"threads_started":1}
I20260927 05:14:08.378614 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.357153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37577 (local address 127.0.0.1:35902)
0927 05:14:08.368880 (+ 11727us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.368897 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.369010 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.371393 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.371397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.371420 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:08.371928 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.371940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.374070 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.374077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.375642 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.375652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.377814 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.377820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.377977 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.378026 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.378088 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.378202 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11588,"mutex_wait_us":24,"thread_start_us":13109,"threads_started":1}
I20260927 05:14:08.378859 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.357573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40333 (local address 127.0.0.1:44368)
0927 05:14:08.370458 (+ 12885us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.370475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.370640 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.371421 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.371427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.371456 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:08.372006 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.372021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.373822 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.373827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.376061 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.376072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.378047 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.378053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.378201 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.378229 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.378313 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.378430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12782,"mutex_wait_us":30,"thread_start_us":11028,"threads_started":1}
I20260927 05:14:08.379585 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.357551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.0.97.1:37577)
0927 05:14:08.370889 (+ 13338us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.370895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.370916 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:14:08.370988 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.370993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.371215 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.371419 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:08.372125 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.373937 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.375781 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.377618 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.378958 (+ 1340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.379198 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.379251 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.379360 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13202,"thread_start_us":13045,"threads_started":1}
I20260927 05:14:08.380240 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.358129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.97.2:40333)
0927 05:14:08.370069 (+ 11940us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.370077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.370555 (+ 478us) server_negotiation.cc:407] Connection header received
0927 05:14:08.370802 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.370810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.371069 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.371209 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:08.372225 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.373687 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.376201 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.377595 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.377731 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.377918 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.379808 (+ 1890us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.379969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11790,"thread_start_us":11685,"threads_started":1}
I20260927 05:14:08.380950 11112 tablet_service.cc:1467] Processing CreateTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 (DEFAULT_TABLE table=test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:08.382805 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c6bf57d0954bb5a5cf271fce999b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:08.382805 10845 tablet_service.cc:1467] Processing CreateTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 (DEFAULT_TABLE table=test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:08.383473 10979 tablet_service.cc:1467] Processing CreateTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 (DEFAULT_TABLE table=test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:08.384188 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c6bf57d0954bb5a5cf271fce999b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:08.384665 10979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c6bf57d0954bb5a5cf271fce999b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:08.395741 11195 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Bootstrap starting.
I20260927 05:14:08.398175 11197 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Bootstrap starting.
I20260927 05:14:08.399032 11195 tablet_bootstrap.cc:654] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:08.400390 11195 log.cc:824] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:08.401705 11197 tablet_bootstrap.cc:654] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:08.402504 11196 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Bootstrap starting.
I20260927 05:14:08.403115 11197 log.cc:824] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:08.405915 11196 tablet_bootstrap.cc:654] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:08.407132 11196 log.cc:824] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:08.408598 11195 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: No bootstrap required, opened a new log
I20260927 05:14:08.408603 11197 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: No bootstrap required, opened a new log
I20260927 05:14:08.408854 11195 ts_tablet_manager.cc:1397] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260927 05:14:08.408870 11197 ts_tablet_manager.cc:1397] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 05:14:08.409914 11196 tablet_bootstrap.cc:492] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: No bootstrap required, opened a new log
I20260927 05:14:08.410145 11196 ts_tablet_manager.cc:1397] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 05:14:08.415961 11195 raft_consensus.cc:348] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.416330 11195 raft_consensus.cc:374] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:08.416433 11195 raft_consensus.cc:729] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dedf2e42a24572a577127fa8451c8c, State: Initialized, Role: FOLLOWER
I20260927 05:14:08.416994 11195 consensus_queue.cc:260] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.418648 11195 ts_tablet_manager.cc:1428] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 05:14:08.419126 11197 raft_consensus.cc:348] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.419531 11197 raft_consensus.cc:374] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:08.419620 11197 raft_consensus.cc:729] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d038c36be54a56a338e321b0c0f4ae, State: Initialized, Role: FOLLOWER
I20260927 05:14:08.420154 11197 consensus_queue.cc:260] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.420316 11196 raft_consensus.cc:348] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.420789 11196 raft_consensus.cc:374] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:08.420940 11196 raft_consensus.cc:729] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144bfcdc4d0447e599dc913b774ad103, State: Initialized, Role: FOLLOWER
I20260927 05:14:08.421204 11044 heartbeater.cc:503] Master 127.0.97.62:43671 was elected leader, sending a full tablet report...
I20260927 05:14:08.421602 11196 consensus_queue.cc:260] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.421808 11197 ts_tablet_manager.cc:1428] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 05:14:08.422659 11177 heartbeater.cc:503] Master 127.0.97.62:43671 was elected leader, sending a full tablet report...
I20260927 05:14:08.424096 11196 ts_tablet_manager.cc:1428] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260927 05:14:08.534972 11178 tablet.cc:2369] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:14:08.559417 11045 tablet.cc:2369] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:14:08.568791 10911 tablet.cc:2369] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:14:08.709347 11202 raft_consensus.cc:482] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:14:08.709709 11202 raft_consensus.cc:504] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.711315 11202 leader_election.cc:288] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 144bfcdc4d0447e599dc913b774ad103 (127.0.97.3:38977), 57dedf2e42a24572a577127fa8451c8c (127.0.97.1:37577)
I20260927 05:14:08.717877 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.711805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52271 (local address 127.0.97.3:38977)
0927 05:14:08.711985 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.711989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.712599 (+ 610us) server_negotiation.cc:407] Connection header received
0927 05:14:08.712796 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.712801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.712909 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.713023 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:08.713594 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.714456 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.716238 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.716989 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.717141 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.717248 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.717511 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.717630 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:14:08.717973 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.711801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38977 (local address 127.0.97.2:52271)
0927 05:14:08.712488 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.712508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.712643 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.713080 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.713085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.713117 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:08.713452 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.713460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.714585 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.714592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.716082 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.716093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.717192 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.717198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.717352 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.717376 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.717490 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.717619 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":379,"threads_started":1}
I20260927 05:14:08.718154 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.712746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46661 (local address 127.0.97.1:37577)
0927 05:14:08.712994 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.713000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.713352 (+ 352us) server_negotiation.cc:407] Connection header received
0927 05:14:08.713548 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.713552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.713606 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.713699 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:08.714171 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.714848 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.716498 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.717248 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.717400 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.717510 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.717797 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.717917 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:14:08.718222 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.712596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37577 (local address 127.0.97.2:46661)
0927 05:14:08.713251 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:08.713269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:08.713408 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:08.713714 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:08.713718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:08.713741 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:08.714035 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.714043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.715021 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.715026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.716348 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:08.716359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.717471 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.717477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:08.717636 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.717658 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:14:08.717771 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 05:14:08.717886 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":364,"threads_started":1}
I20260927 05:14:08.719197 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c6bf57d0954bb5a5cf271fce999b24" candidate_uuid: "c7d038c36be54a56a338e321b0c0f4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144bfcdc4d0447e599dc913b774ad103" is_pre_election: true
I20260927 05:14:08.719195 10865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c6bf57d0954bb5a5cf271fce999b24" candidate_uuid: "c7d038c36be54a56a338e321b0c0f4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57dedf2e42a24572a577127fa8451c8c" is_pre_election: true
I20260927 05:14:08.719691 11132 raft_consensus.cc:2454] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7d038c36be54a56a338e321b0c0f4ae in term 0.
I20260927 05:14:08.719764 10865 raft_consensus.cc:2454] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7d038c36be54a56a338e321b0c0f4ae in term 0.
I20260927 05:14:08.720330 10935 leader_election.cc:302] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144bfcdc4d0447e599dc913b774ad103, c7d038c36be54a56a338e321b0c0f4ae; no voters:
I20260927 05:14:08.720713 11202 raft_consensus.cc:2781] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:14:08.720850 11202 raft_consensus.cc:482] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:14:08.720955 11202 raft_consensus.cc:3037] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:08.723097 11202 raft_consensus.cc:504] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.723671 11202 leader_election.cc:288] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [CANDIDATE]: Term 1 election: Requested vote from peers 144bfcdc4d0447e599dc913b774ad103 (127.0.97.3:38977), 57dedf2e42a24572a577127fa8451c8c (127.0.97.1:37577)
I20260927 05:14:08.723985 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c6bf57d0954bb5a5cf271fce999b24" candidate_uuid: "c7d038c36be54a56a338e321b0c0f4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144bfcdc4d0447e599dc913b774ad103"
I20260927 05:14:08.724053 10865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c6bf57d0954bb5a5cf271fce999b24" candidate_uuid: "c7d038c36be54a56a338e321b0c0f4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57dedf2e42a24572a577127fa8451c8c"
I20260927 05:14:08.724167 11132 raft_consensus.cc:3037] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:08.724243 10865 raft_consensus.cc:3037] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:08.726740 11132 raft_consensus.cc:2454] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7d038c36be54a56a338e321b0c0f4ae in term 1.
I20260927 05:14:08.727126 10935 leader_election.cc:302] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144bfcdc4d0447e599dc913b774ad103, c7d038c36be54a56a338e321b0c0f4ae; no voters:
I20260927 05:14:08.727296 10865 raft_consensus.cc:2454] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7d038c36be54a56a338e321b0c0f4ae in term 1.
I20260927 05:14:08.727414 11202 raft_consensus.cc:2781] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:08.727883 11202 raft_consensus.cc:686] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 1 LEADER]: Becoming Leader. State: Replica: c7d038c36be54a56a338e321b0c0f4ae, State: Running, Role: LEADER
I20260927 05:14:08.728607 11202 consensus_queue.cc:237] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } }
I20260927 05:14:08.732744 10721 catalog_manager.cc:5495] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae reported cstate change: term changed from 0 to 1, leader changed from <none> to c7d038c36be54a56a338e321b0c0f4ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c7d038c36be54a56a338e321b0c0f4ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7d038c36be54a56a338e321b0c0f4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40333 } health_report { overall_health: HEALTHY } } }
I20260927 05:14:08.825837 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.821273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.97.2:40333)
0927 05:14:08.821449 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.821454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.821473 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:14:08.821542 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.821545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.821615 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.821728 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:14:08.822028 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.822712 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.823992 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.824373 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.824473 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.824851 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:14:08.825293 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:14:08.825394 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.825477 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.825535 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:14:08.833279 11132 raft_consensus.cc:1261] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 1 FOLLOWER]: Refusing update from remote peer c7d038c36be54a56a338e321b0c0f4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:14:08.833223 10865 raft_consensus.cc:1261] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 1 FOLLOWER]: Refusing update from remote peer c7d038c36be54a56a338e321b0c0f4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:14:08.834156 11206 consensus_queue.cc:1035] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "144bfcdc4d0447e599dc913b774ad103" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:14:08.834586 11202 consensus_queue.cc:1035] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "57dedf2e42a24572a577127fa8451c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:14:08.847240 11211 mvcc.cc:205] Tried to move back new op lower bound from 7333830856012222464 to 7333830855595950080. Current Snapshot: MvccSnapshot[applied={T|T < 7333830856012222464}]
I20260927 05:14:08.862723 10721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:14:08.863010 10721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:14:08.866137 10721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5aa2e18a51c454abc6502911cedb080: Sending DeleteTablet for 3 replicas of tablet 36c6bf57d0954bb5a5cf271fce999b24
I20260927 05:14:08.867178 11112 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36848
I20260927 05:14:08.867497 10979 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44368
I20260927 05:14:08.867763 10845 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c6bf57d0954bb5a5cf271fce999b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35902
I20260927 05:14:08.871079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10782
I20260927 05:14:08.879789 11221 tablet_replica.cc:330] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: stopping tablet replica
I20260927 05:14:08.880242 11221 raft_consensus.cc:2229] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:14:08.880617 11219 tablet_replica.cc:330] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: stopping tablet replica
I20260927 05:14:08.880775 11221 raft_consensus.cc:2258] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:08.881014 11219 raft_consensus.cc:2229] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:14:08.881458 11219 raft_consensus.cc:2258] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:08.882213 11220 tablet_replica.cc:330] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: stopping tablet replica
I20260927 05:14:08.882623 11220 raft_consensus.cc:2229] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:14:08.883394 11220 raft_consensus.cc:2258] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:08.883987 11221 ts_tablet_manager.cc:1905] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:14:08.885010 11219 ts_tablet_manager.cc:1905] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:14:08.885557 11220 ts_tablet_manager.cc:1905] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:14:08 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:14:08.891795 11219 ts_tablet_manager.cc:1918] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:08.891985 11219 log.cc:1196] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/36c6bf57d0954bb5a5cf271fce999b24
I20260927 05:14:08.892164 11221 ts_tablet_manager.cc:1918] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:08.892514 11219 ts_tablet_manager.cc:1939] T 36c6bf57d0954bb5a5cf271fce999b24 P 144bfcdc4d0447e599dc913b774ad103: Deleting consensus metadata
I20260927 05:14:08.894340 10708 catalog_manager.cc:4841] TS 144bfcdc4d0447e599dc913b774ad103 (127.0.97.3:38977): tablet 36c6bf57d0954bb5a5cf271fce999b24 (table test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]) successfully deleted
I20260927 05:14:08.894899 11221 log.cc:1196] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/36c6bf57d0954bb5a5cf271fce999b24
I20260927 05:14:08.895484 11221 ts_tablet_manager.cc:1939] T 36c6bf57d0954bb5a5cf271fce999b24 P 57dedf2e42a24572a577127fa8451c8c: Deleting consensus metadata
I20260927 05:14:08.897166 10708 catalog_manager.cc:4841] TS 57dedf2e42a24572a577127fa8451c8c (127.0.97.1:37577): tablet 36c6bf57d0954bb5a5cf271fce999b24 (table test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]) successfully deleted
I20260927 05:14:08.899421 11220 ts_tablet_manager.cc:1918] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:08.899636 11220 log.cc:1196] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/36c6bf57d0954bb5a5cf271fce999b24
I20260927 05:14:08.899843 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:08.872512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.97.1:37577)
0927 05:14:08.872704 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:08.872710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:08.880007 (+ 7297us) server_negotiation.cc:407] Connection header received
0927 05:14:08.880181 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:08.880187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:08.880290 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:08.880431 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:08.883210 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.884329 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.886320 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:08.886984 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:08.887120 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:08.892502 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:08.892516 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:08.892532 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:08.892574 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:08.896667 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.897557 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.897565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.897569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.897719 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:08.898147 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:08.898153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:08.898156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:08.898567 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:08.898849 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:08.899218 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 05:14:08.899446 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:14:08.900159 11220 ts_tablet_manager.cc:1939] T 36c6bf57d0954bb5a5cf271fce999b24 P c7d038c36be54a56a338e321b0c0f4ae: Deleting consensus metadata
I20260927 05:14:08.900619 10905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:08.901937 10707 catalog_manager.cc:4841] TS c7d038c36be54a56a338e321b0c0f4ae (127.0.97.2:40333): tablet 36c6bf57d0954bb5a5cf271fce999b24 (table test-table [id=b86317165abe40f8b1fa9b95eb9d30f1]) successfully deleted
I20260927 05:14:08.989331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10782
I20260927 05:14:09.013254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10916
Sep 27 05:14:09 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:14:09.029721 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:09.014255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.97.2:40333)
0927 05:14:09.014416 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:09.014421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:09.014436 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:14:09.014487 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:09.014490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:09.014549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:09.014626 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:09.016208 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:09.017075 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:09.018860 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:09.019406 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:09.019497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:09.024145 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:09.024156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:09.024171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:09.024218 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:09.027102 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:09.027747 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:09.027754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:09.027756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:09.027905 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:09.028263 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:09.028268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:09.028273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:09.028605 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:09.028953 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:09.029163 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:14:09.029360 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:14:09.030385 11039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:09.138707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10916
I20260927 05:14:09.156526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 11049
Sep 27 05:14:09 dist-test-slave-1t6p krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:14:09.170176 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:09.157252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.97.3:38977)
0927 05:14:09.157426 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:09.157430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:09.157448 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:14:09.157592 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:09.157596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:09.157656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:09.157736 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:09.158950 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:09.159695 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:09.160809 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:09.161166 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:09.161250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:09.165076 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:09.165088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:09.165103 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:09.165137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:09.167948 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:09.168610 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:09.168616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:09.168619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:09.168741 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:09.169031 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:09.169034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:09.169036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:09.169348 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:09.169556 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:09.169767 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 05:14:09.169909 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:14:09.170753 11172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:09.258160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 11049
I20260927 05:14:09.276077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 10689
I20260927 05:14:09.277163 10751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:09.379611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 10689
2026-09-27T05:14:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:14:09 dist-test-slave-1t6p krb5kdc[11229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:14:09 dist-test-slave-1t6p krb5kdc[11229](info): set up 2 sockets
Sep 27 05:14:09 dist-test-slave-1t6p krb5kdc[11229](info): commencing operation
krb5kdc: starting...
W20260927 05:14:10.471882 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:14:10 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:14:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 05:14:10.761771 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7OtwtO/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/postgres -l logfile start
2026-09-27 05:14:13.147 UTC [11259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:14:13.147 UTC [11259] LOG: listening on IPv4 address "127.0.97.20", port 48807
2026-09-27 05:14:13.151 UTC [11259] LOG: listening on Unix socket "/tmp/.s.PGSQL.48807"
2026-09-27 05:14:13.173 UTC [11262] LOG: database system was shut down at 2026-09-27 05:14:11 UTC
2026-09-27 05:14:13.178 UTC [11259] LOG: database system is ready to accept connections
I20260927 05:14:14.143512 388 mini_postgres.cc:96] Postgres bound to 48807
2026-09-27 05:14:14.159 UTC [11270] FATAL: database "slave" does not exist
127.0.97.20:48807 - accepting connections
I20260927 05:14:14.160468 388 mini_ranger.cc:161] Starting Ranger...
I20260927 05:14:14.183339 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:14:14.650725 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:14:14.650861 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin
2026-09-27 05:14:14,928 [I] DB FLAVOR :POSTGRES
2026-09-27 05:14:14,929 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:14:14,929 [I] Checking connection..
2026-09-27 05:14:14,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:14:15,158 [I] Checking connection passed.
2026-09-27 05:14:15,159 [I] --------- Verifying version history table ---------
2026-09-27 05:14:15,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:14:15,410 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:14:15,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:14:15,670 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:14:15,670 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:14:15,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:14:15.922 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 05:14:15,928 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:14:15,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:14:16,181 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:14:16,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:14:16,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:14:16,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:14:16,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1t6p.c.gcp-upstream.internal','N') ;"
2026-09-27 05:14:16,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:14:17,130 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:14:17,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:14:17,374 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:14:17,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:14:17,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:14:17,822 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:14:17,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:14:18.736 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.752 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.768 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.783 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.798 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.817 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.878 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.885 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.899 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.909 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.918 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.932 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.941 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.947 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.955 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:18.962 UTC [11442] WARNING: there is no transaction in progress
2026-09-27 05:14:19,600 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:14:19,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:14:19,821 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:14:19,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:14:20,059 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:14:20,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:14:20,302 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:14:20,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:14:20,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:14:20,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1t6p.c.gcp-upstream.internal';"
2026-09-27 05:14:21,001 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:14:21,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:14:21,231 [I] DB_PATCHES have already been applied
I20260927 05:14:21.237087 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7OtwtO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:14:21.237191 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 05:14:21.253739 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:46703
Sep 27, 2026 5:14:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:14:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:14:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:14:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:14:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:14:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:14:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:14:22.383535 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.223s user 0.005s sys 0.031s
Sep 27, 2026 5:14:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:14:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 5:14:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:14:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:14:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:14:44.128823 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:14:44 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:14:44.309017 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:14:44.327147 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:14:45.188941 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:14:45.189047 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms
I20260927 05:14:45.189193 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48807
2026-09-27 05:14:45,597 [I] DB FLAVOR :POSTGRES
2026-09-27 05:14:45,598 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:14:45,598 [I] Checking connection
2026-09-27 05:14:45,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:14:45,841 [I] connection success
2026-09-27 05:14:45,841 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:14:45,841 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:14:45,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:14:46,096 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:14:46,097 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:14:46,097 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:14:46,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:14:46,365 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:14:46.785293 388 mini_ranger_kms.cc:310] Created kms service
I20260927 05:14:46.948158 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:14:46.991238 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:14:46 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:14:47.376850 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:14:47.377190 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:14:47.377256 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 05:14:47.397444 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11622
I20260927 05:14:47.397616 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:33875
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 05:14:49 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790486089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:49.686828 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.378s user 0.115s sys 0.297s
I20260927 05:14:49.687080 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:14:49.687280 388 mini_ranger_kms.cc:401] 127.0.97.20:33875/kms/v1/keys
Sep 27 05:14:49 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:14:50 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:14:52 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:14:52.552809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45363
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45363
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33875/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:14:52.888499 11673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:52.888784 11673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:52.888918 11673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:52.895881 11673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:14:52.895973 11673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:52.896049 11673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:52.896147 11673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:14:52.896236 11673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:14:52.906157 11673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:33875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45363
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:52.907442 11673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:52.908668 11673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:52.922132 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:52.922251 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:52.922778 11673 server_base.cc:1034] running on GCE node
W20260927 05:14:52.922950 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:52.923971 11673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:52.925645 11673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:52.926884 11673 hybrid_clock.cc:648] HybridClock initialized: now 1790486092926839 us; error 53 us; skew 500 ppm
I20260927 05:14:52.927290 11673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:52 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:52.934091 11673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:14:52.935701 11673 webserver.cc:460] Webserver started at http://127.0.97.62:39955/ using document root <none> and password file <none>
I20260927 05:14:52.936331 11673 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:52.936468 11673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:52.936758 11673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:14:52 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:14:52.987589 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "60ac3e1a574441c9a99b0f9adb272602"
format_stamp: "Formatted at 2026-09-27 05:14:52 on dist-test-slave-1t6p"
server_key: "28bc3f1da3933fd298cf4e8f8d6a3c3e"
server_key_iv: "4b9cfae35e69466c1542faa91c8b2ee7"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:52.988627 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "60ac3e1a574441c9a99b0f9adb272602"
format_stamp: "Formatted at 2026-09-27 05:14:52 on dist-test-slave-1t6p"
server_key: "28bc3f1da3933fd298cf4e8f8d6a3c3e"
server_key_iv: "4b9cfae35e69466c1542faa91c8b2ee7"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:52.995424 11673 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 05:14:53.015331 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:53.017021 11673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:14:53.017213 11673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "60ac3e1a574441c9a99b0f9adb272602"
format_stamp: "Formatted at 2026-09-27 05:14:52 on dist-test-slave-1t6p"
server_key: "28bc3f1da3933fd298cf4e8f8d6a3c3e"
server_key_iv: "4b9cfae35e69466c1542faa91c8b2ee7"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:53.017450 11673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:53.035418 11673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:53.040444 11673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:53.040781 11673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:53.060999 11673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45363
I20260927 05:14:53.061002 11739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45363 every 8 connection(s)
I20260927 05:14:53.062988 11673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:14:53.066867 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:53.066972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 11673
I20260927 05:14:53.067191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:14:53.067991 388 external_mini_cluster.cc:1428] Setting key 0296153789b915f8b2e564a5a7401614
I20260927 05:14:53.084690 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: Bootstrap starting.
I20260927 05:14:53.088233 11740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:53.089612 11740 log.cc:824] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:53.092742 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: No bootstrap required, opened a new log
Sep 27 05:14:53 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:53.102360 11740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } }
I20260927 05:14:53.102821 11740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:53.102931 11740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ac3e1a574441c9a99b0f9adb272602, State: Initialized, Role: FOLLOWER
I20260927 05:14:53.103609 11740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } }
I20260927 05:14:53.103894 11740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:14:53.104033 11740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:14:53.104220 11740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:53.107153 11740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } }
I20260927 05:14:53.107841 11740 leader_election.cc:302] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ac3e1a574441c9a99b0f9adb272602; no voters:
I20260927 05:14:53.108500 11740 leader_election.cc:288] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:14:53.108616 11745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:53.109012 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:53.070003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.97.62:45363)
0927 05:14:53.084927 (+ 14924us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:53.084943 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:53.084992 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:14:53.087649 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:53.087678 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:53.088159 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:53.088568 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:53.090282 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:53.092084 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:53.093848 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:53.094935 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:53.095163 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:53.099805 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:53.099820 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:53.099840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:53.099887 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:53.104206 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:53.105046 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:53.105055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:53.105063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:53.105233 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:53.105668 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:53.105673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:53.105676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:53.106365 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:53.106616 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:53.107152 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 05:14:53.107555 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14702,"thread_start_us":14479,"threads_started":1}
I20260927 05:14:53.114989 11740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:14:53.117199 11745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [term 1 LEADER]: Becoming Leader. State: Replica: 60ac3e1a574441c9a99b0f9adb272602, State: Running, Role: LEADER
I20260927 05:14:53.117743 11745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } }
I20260927 05:14:53.126791 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ac3e1a574441c9a99b0f9adb272602. Latest consensus state: current_term: 1 leader_uuid: "60ac3e1a574441c9a99b0f9adb272602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } } }
I20260927 05:14:53.127517 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:53.126765 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ac3e1a574441c9a99b0f9adb272602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ac3e1a574441c9a99b0f9adb272602" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45363 } } }
I20260927 05:14:53.127923 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602 [sys.catalog]: This master's current role is: LEADER
I20260927 05:14:53.128954 11759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:14:53.139065 11759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:14:53.147912 11740 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:14:53.161188 11759 catalog_manager.cc:1347] Generated new cluster ID: f891b9380a6a460e9a4a1dbadd5e18c1
I20260927 05:14:53.161398 11759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:14:53.203967 11759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:14:53.206180 11759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:14:53.216974 11759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: Generated new TSK 0
I20260927 05:14:53.218119 11759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:14:54.056993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:14:54 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:14:54.459069 11786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:54.459424 11786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:54.459625 11786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:54.471443 11786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:54.471565 11786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:54.471783 11786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:14:54.487385 11786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:33875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:54.489184 11786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:54.490883 11786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:54.525296 11797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:54.525826 11796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:54.526656 11799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:54.527076 11786 server_base.cc:1034] running on GCE node
I20260927 05:14:54.527833 11786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:54.530581 11786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:54.531895 11786 hybrid_clock.cc:648] HybridClock initialized: now 1790486094531841 us; error 80 us; skew 500 ppm
I20260927 05:14:54.532246 11786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:54 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:54.535447 11786 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:14:54.540438 11786 webserver.cc:460] Webserver started at http://127.0.97.1:46077/ using document root <none> and password file <none>
I20260927 05:14:54.541110 11786 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:54.541241 11786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:54.541558 11786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:14:54 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:14:54.562698 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47"
format_stamp: "Formatted at 2026-09-27 05:14:54 on dist-test-slave-1t6p"
server_key: "59e6ae3c9dae90d2c8ef0c7de2f8c801"
server_key_iv: "4effc256c32179aceee1e7b57e94661d"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:54.564182 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47"
format_stamp: "Formatted at 2026-09-27 05:14:54 on dist-test-slave-1t6p"
server_key: "59e6ae3c9dae90d2c8ef0c7de2f8c801"
server_key_iv: "4effc256c32179aceee1e7b57e94661d"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:54.580323 11786 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.010s sys 0.005s
I20260927 05:14:54.611707 11806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:54.614928 11786 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.005s sys 0.004s
I20260927 05:14:54.615206 11786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47"
format_stamp: "Formatted at 2026-09-27 05:14:54 on dist-test-slave-1t6p"
server_key: "59e6ae3c9dae90d2c8ef0c7de2f8c801"
server_key_iv: "4effc256c32179aceee1e7b57e94661d"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:54.615545 11786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:54.641548 11786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:54.646868 11786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:54.647217 11786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:54.648380 11786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:54.650102 11786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:54.650202 11786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:54.650326 11786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:54.650391 11786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:54.699331 11786 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44391
I20260927 05:14:54.699352 11918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44391 every 8 connection(s)
I20260927 05:14:54.701076 11786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:14:54.707136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 11786
I20260927 05:14:54.707326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:14:54.708078 388 external_mini_cluster.cc:1428] Setting key 73cc8416b784baf8e2c52657c8d2e22b
Sep 27 05:14:54 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:14:54.744009 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:54.708788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39657 (local address 127.0.97.62:45363)
0927 05:14:54.719168 (+ 10380us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:54.719176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:54.719196 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:14:54.719258 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:54.719265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:54.719356 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:54.719460 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:54.721616 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:54.722901 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:54.725317 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:54.725939 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:54.726059 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:54.734257 (+ 8198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:54.734264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:54.734270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:54.734319 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:54.737862 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:54.738788 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:54.738794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:54.738798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:54.738937 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:54.739477 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:54.739482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:54.739486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:54.739814 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:54.739939 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:54.742852 (+ 2913us) server_negotiation.cc:299] Negotiation successful
0927 05:14:54.743107 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10235,"thread_start_us":11716,"threads_started":1}
I20260927 05:14:54.744315 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:54.706226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45363 (local address 127.0.97.1:39657)
0927 05:14:54.715549 (+ 9323us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:54.715607 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:54.718470 (+ 2863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:54.719926 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:54.719942 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:54.720599 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:54.721424 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:54.721452 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:54.723132 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:54.723138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:54.725169 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:54.725185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:54.726189 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:54.726196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:54.726493 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:54.731226 (+ 4733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:54.731256 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:54.733999 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:54.738081 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:54.738090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:54.738107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:54.738648 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:54.739091 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:54.739097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:54.739101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:54.739346 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:54.740020 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:54.740027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:54.740453 (+ 426us) client_negotiation.cc:769] Sending connection context
0927 05:14:54.740742 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 05:14:54.741257 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9031,"thread_start_us":8825,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:14:54.746152 11919 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45363
I20260927 05:14:54.746471 11919 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:54.747337 11919 heartbeater.cc:511] Master 127.0.97.62:45363 requested a full tablet report, sending...
I20260927 05:14:54.749670 11705 ts_manager.cc:194] Registered new tserver with Master: 6be73b83c4dc4b81ba9b04f5a2d47d47 (127.0.97.1:44391)
I20260927 05:14:54.752089 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:14:54.782459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:14:54 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 05:14:55.233702 11925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:55.234174 11925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:55.234366 11925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:55.245428 11925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:55.245635 11925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:55.245859 11925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:14:55.260996 11925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:33875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:55.262909 11925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:55.264722 11925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:55.289323 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:55.289350 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:55.294143 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:55.294781 11925 server_base.cc:1034] running on GCE node
I20260927 05:14:55.295526 11925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:55.296685 11925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:55.297961 11925 hybrid_clock.cc:648] HybridClock initialized: now 1790486095297895 us; error 49 us; skew 500 ppm
I20260927 05:14:55.298395 11925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:55.303069 11925 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:14:55.308439 11925 webserver.cc:460] Webserver started at http://127.0.97.2:42163/ using document root <none> and password file <none>
I20260927 05:14:55.309350 11925 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:55.309487 11925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:55.309861 11925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:14:55.329344 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "af8d20394fe3419786f361d0496e1039"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d2eef007cce98c4728d1ad1e6fef489f"
server_key_iv: "23e45153c473ce7fbe780e4ee391aea9"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.330420 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af8d20394fe3419786f361d0496e1039"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d2eef007cce98c4728d1ad1e6fef489f"
server_key_iv: "23e45153c473ce7fbe780e4ee391aea9"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.344940 11925 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.012s
I20260927 05:14:55.371788 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:55.373787 11925 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.008s
I20260927 05:14:55.374054 11925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "af8d20394fe3419786f361d0496e1039"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d2eef007cce98c4728d1ad1e6fef489f"
server_key_iv: "23e45153c473ce7fbe780e4ee391aea9"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.374397 11925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:55.418671 11925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:55.423746 11925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:55.424474 11925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:55.425576 11925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:55.428217 11925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:55.428359 11925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 05:14:55.428527 11925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:55.428632 11925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:55.489852 11925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36193
I20260927 05:14:55.489909 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36193 every 8 connection(s)
I20260927 05:14:55.491222 11925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:14:55.494478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 11925
I20260927 05:14:55.494678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:14:55.495227 388 external_mini_cluster.cc:1428] Setting key f8c4da2de6c3a66d02fb873445c562b5
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:14:55.524407 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:55.493975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41757 (local address 127.0.97.62:45363)
0927 05:14:55.506944 (+ 12969us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:55.506952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:55.509806 (+ 2854us) server_negotiation.cc:407] Connection header received
0927 05:14:55.509863 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:55.509866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:55.509937 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:55.510060 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:55.511601 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:55.512335 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:55.513898 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:55.514292 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:55.514427 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:55.517970 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:55.517976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:55.517979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:55.518016 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:55.520914 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:55.521732 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:55.521737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:55.521741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:55.521910 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:55.522427 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:55.522434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:55.522439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:55.522779 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:55.522977 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:55.523811 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 05:14:55.524037 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12377,"thread_start_us":12265,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:14:55.525558 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:55.494648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45363 (local address 127.0.97.2:41757)
0927 05:14:55.507655 (+ 13007us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:55.507713 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:55.509512 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:55.510329 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:55.510337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:55.510772 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:55.511432 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:55.511450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:55.512502 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:55.512507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:55.513742 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:55.513753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:55.514522 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:55.514529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:55.514745 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:55.515487 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:55.515523 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:55.517754 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:55.521118 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:55.521127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:55.521149 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:55.521609 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:55.522068 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:55.522076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:55.522080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:55.522305 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:55.522927 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:55.522937 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:55.523361 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 05:14:55.523690 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 05:14:55.524388 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12706,"thread_start_us":12427,"threads_started":1}
I20260927 05:14:55.527235 12062 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45363
I20260927 05:14:55.527546 12062 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:55.528190 12062 heartbeater.cc:511] Master 127.0.97.62:45363 requested a full tablet report, sending...
I20260927 05:14:55.529587 11705 ts_manager.cc:194] Registered new tserver with Master: af8d20394fe3419786f361d0496e1039 (127.0.97.2:36193)
I20260927 05:14:55.530355 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:14:55.556427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:14:55.755785 11919 heartbeater.cc:503] Master 127.0.97.62:45363 was elected leader, sending a full tablet report...
W20260927 05:14:55.904389 12068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:14:55.904691 12068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:14:55.904860 12068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:14:55.911952 12068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:14:55.912087 12068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:14:55.912266 12068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:14:55.922595 12068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:33875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:14:55.923954 12068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:14:55.925207 12068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:14:55.937675 12073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:55.937739 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:14:55.937700 12076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:14:55.938516 12068 server_base.cc:1034] running on GCE node
I20260927 05:14:55.939155 12068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:14:55.939966 12068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:14:55.941164 12068 hybrid_clock.cc:648] HybridClock initialized: now 1790486095941111 us; error 50 us; skew 500 ppm
I20260927 05:14:55.941457 12068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:14:55.944557 12068 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:14:55.945869 12068 webserver.cc:460] Webserver started at http://127.0.97.3:45567/ using document root <none> and password file <none>
I20260927 05:14:55.946489 12068 fs_manager.cc:359] Metadata directory not provided
I20260927 05:14:55.946622 12068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:14:55.946915 12068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:14:55 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:14:55.962260 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "a4da1f3d82ff48bd893e92c0059f8a55"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d4f356eb6cdaee1ee156b84b4d8f015c"
server_key_iv: "ead2ebd0051cd839051dcce541b1f03b"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.963332 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4da1f3d82ff48bd893e92c0059f8a55"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d4f356eb6cdaee1ee156b84b4d8f015c"
server_key_iv: "ead2ebd0051cd839051dcce541b1f03b"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.969732 12068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:14:55.987576 12083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:55.989313 12068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 05:14:55.989488 12068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "a4da1f3d82ff48bd893e92c0059f8a55"
format_stamp: "Formatted at 2026-09-27 05:14:55 on dist-test-slave-1t6p"
server_key: "d4f356eb6cdaee1ee156b84b4d8f015c"
server_key_iv: "ead2ebd0051cd839051dcce541b1f03b"
server_key_version: "kuduclusterkey@0"
I20260927 05:14:55.989724 12068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:14:56.007622 12068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:14:56.011806 12068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:14:56.012046 12068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:14:56.012948 12068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:14:56.014250 12068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:14:56.014326 12068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:56.014411 12068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:14:56.014454 12068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:14:56.049715 12068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45935
I20260927 05:14:56.049721 12195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45935 every 8 connection(s)
I20260927 05:14:56.050966 12068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:14:56.059984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 12068
I20260927 05:14:56.060163 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:14:56.060606 388 external_mini_cluster.cc:1428] Setting key fed97cc146f0c434cb7c926167a52b76
Sep 27 05:14:56 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:14:56.071396 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.053450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48799 (local address 127.0.97.62:45363)
0927 05:14:56.053900 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.053905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.054794 (+ 889us) server_negotiation.cc:407] Connection header received
0927 05:14:56.056470 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.056476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.056561 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.056679 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:56.058027 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.058770 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.060440 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.060950 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.061078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.065227 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:56.065232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:56.065235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:56.065263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:56.068619 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.069290 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.069295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.069299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.069447 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.069834 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.069839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.069843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.070177 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:56.070319 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.070945 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.071157 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
I20260927 05:14:56.072300 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.053982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45363 (local address 127.0.97.3:48799)
0927 05:14:56.054633 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.054674 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.056214 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.056912 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.056920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.057318 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:14:56.057868 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.057884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.058925 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.058932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.060290 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.060301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.061162 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.061170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.061462 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.062179 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:14:56.062208 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:14:56.065010 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:14:56.068750 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:56.068757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:56.068768 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:56.069165 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:56.069543 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:14:56.069546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:14:56.069548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:14:56.069728 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:14:56.070330 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:14:56.070336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:14:56.070613 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.070844 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.071264 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260927 05:14:56.073575 12196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45363
I20260927 05:14:56.073822 12196 heartbeater.cc:464] Registering TS with master...
I20260927 05:14:56.074443 12196 heartbeater.cc:511] Master 127.0.97.62:45363 requested a full tablet report, sending...
I20260927 05:14:56.076465 11705 ts_manager.cc:194] Registered new tserver with Master: a4da1f3d82ff48bd893e92c0059f8a55 (127.0.97.3:45935)
I20260927 05:14:56.077401 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48799
I20260927 05:14:56.088905 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:14:56.106288 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.093703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.0.97.62:45363)
0927 05:14:56.093857 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.093863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.094363 (+ 500us) server_negotiation.cc:407] Connection header received
0927 05:14:56.094492 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.094496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.094568 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.094688 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:56.096972 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.097716 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.099036 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.099461 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.099554 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.101747 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:56.101752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:56.101755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:56.101783 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:56.104285 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.104845 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.104848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.104850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.104960 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.105237 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.105240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.105242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.105475 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:56.105654 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.105922 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.106065 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:14:56.109866 11705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:14:56.116931 11705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:14:56.142829 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.129225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.0.97.2:36193)
0927 05:14:56.129847 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.129858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.130427 (+ 569us) server_negotiation.cc:407] Connection header received
0927 05:14:56.130830 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.130838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.131161 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.131374 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:56.132539 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.134577 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.138650 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.140600 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.140706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.140900 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.142280 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.142450 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":327,"threads_started":1}
I20260927 05:14:56.143560 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.128926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36193 (local address 127.0.0.1:38758)
0927 05:14:56.130272 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.130322 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.130584 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.131756 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.131763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.131801 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:56.132303 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.132319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.134793 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.134802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.138423 (+ 3621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.138443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.141087 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.141093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.141313 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.142073 (+ 760us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.142892 (+ 819us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.143189 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":683,"thread_start_us":952,"threads_started":1}
I20260927 05:14:56.147033 11997 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (DEFAULT_TABLE table=test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:56.148447 11997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:56.149223 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.129203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44391 (local address 127.0.0.1:46290)
0927 05:14:56.140572 (+ 11369us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.140588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.140713 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.141298 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.141306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.141340 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:56.141877 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.141890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.144252 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.144257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.146267 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.146281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.148453 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.148460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.148639 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.148679 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.148789 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.148903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11257,"thread_start_us":13132,"threads_started":1}
I20260927 05:14:56.149345 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.129855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.0.97.1:44391)
0927 05:14:56.130440 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.130448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.140672 (+ 10224us) server_negotiation.cc:407] Connection header received
0927 05:14:56.140852 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.140859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.141091 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.141280 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:56.142069 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.144143 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.146411 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.147516 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.147632 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.148307 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.148842 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.148998 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":743,"threads_started":1}
I20260927 05:14:56.150863 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.130256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45935 (local address 127.0.0.1:58282)
0927 05:14:56.142187 (+ 11931us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.142205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.142296 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.143522 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.143526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.143555 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:56.144150 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.144162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.146298 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.146303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.148415 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.148434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.150145 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.150152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.150319 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.150355 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.150441 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.150565 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11818,"thread_start_us":10478,"threads_started":1}
I20260927 05:14:56.150990 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.130832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.97.3:45935)
0927 05:14:56.142971 (+ 12139us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.142980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.143032 (+ 52us) server_negotiation.cc:407] Connection header received
0927 05:14:56.143113 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.143117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.143312 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.144347 (+ 1035us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:56.144603 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.146140 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.148597 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.149937 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.150065 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.150282 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.150496 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.150641 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11986,"thread_start_us":11850,"threads_started":1}
I20260927 05:14:56.152642 11854 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (DEFAULT_TABLE table=test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:56.154012 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:56.154469 12131 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (DEFAULT_TABLE table=test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:14:56.155788 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:14:56.160390 12215 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Bootstrap starting.
I20260927 05:14:56.163971 12215 tablet_bootstrap.cc:654] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:56.164911 12215 log.cc:824] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:56.169102 12217 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Bootstrap starting.
I20260927 05:14:56.169723 12215 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: No bootstrap required, opened a new log
I20260927 05:14:56.170049 12215 ts_tablet_manager.cc:1397] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 05:14:56.172538 12217 tablet_bootstrap.cc:654] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:56.173302 12218 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Bootstrap starting.
I20260927 05:14:56.174065 12217 log.cc:824] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:56.176321 12218 tablet_bootstrap.cc:654] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Neither blocks nor log segments found. Creating new log.
I20260927 05:14:56.176985 12217 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: No bootstrap required, opened a new log
I20260927 05:14:56.177230 12217 ts_tablet_manager.cc:1397] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:14:56.177408 12218 log.cc:824] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Log is configured to *not* fsync() on all Append() calls
I20260927 05:14:56.179713 12218 tablet_bootstrap.cc:492] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: No bootstrap required, opened a new log
I20260927 05:14:56.179940 12218 ts_tablet_manager.cc:1397] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 05:14:56.181280 12215 raft_consensus.cc:348] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.181685 12215 raft_consensus.cc:374] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:56.181789 12215 raft_consensus.cc:729] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8d20394fe3419786f361d0496e1039, State: Initialized, Role: FOLLOWER
I20260927 05:14:56.182360 12215 consensus_queue.cc:260] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.183430 12062 heartbeater.cc:503] Master 127.0.97.62:45363 was elected leader, sending a full tablet report...
I20260927 05:14:56.184036 12215 ts_tablet_manager.cc:1428] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260927 05:14:56.187974 12217 raft_consensus.cc:348] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.188380 12217 raft_consensus.cc:374] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:56.188485 12217 raft_consensus.cc:729] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be73b83c4dc4b81ba9b04f5a2d47d47, State: Initialized, Role: FOLLOWER
I20260927 05:14:56.188493 12218 raft_consensus.cc:348] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.189011 12218 raft_consensus.cc:374] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:14:56.189109 12218 raft_consensus.cc:729] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4da1f3d82ff48bd893e92c0059f8a55, State: Initialized, Role: FOLLOWER
I20260927 05:14:56.189079 12217 consensus_queue.cc:260] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.189781 12218 consensus_queue.cc:260] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.190809 12217 ts_tablet_manager.cc:1428] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 05:14:56.190811 12196 heartbeater.cc:503] Master 127.0.97.62:45363 was elected leader, sending a full tablet report...
I20260927 05:14:56.191380 12218 ts_tablet_manager.cc:1428] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260927 05:14:56.211429 11920 tablet.cc:2369] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:14:56.243285 12063 tablet.cc:2369] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:14:56.265743 12222 raft_consensus.cc:482] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:14:56.266053 12222 raft_consensus.cc:504] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.267715 12222 leader_election.cc:288] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8d20394fe3419786f361d0496e1039 (127.0.97.2:36193), a4da1f3d82ff48bd893e92c0059f8a55 (127.0.97.3:45935)
I20260927 05:14:56.273681 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.268102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38509 (local address 127.0.97.3:45935)
0927 05:14:56.268305 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.268314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.268724 (+ 410us) server_negotiation.cc:407] Connection header received
0927 05:14:56.268957 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.268962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.269018 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.269108 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:56.269632 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.270404 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.272058 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.272815 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.272964 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.273063 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.273327 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.273444 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 05:14:56.273774 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.267965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45935 (local address 127.0.97.1:38509)
0927 05:14:56.268616 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.268632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.268808 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.269137 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.269142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.269167 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:56.269495 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.269503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.270527 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.270535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.271908 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.271920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.273013 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.273020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.273172 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.273194 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.273303 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.273423 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":353,"threads_started":1}
I20260927 05:14:56.274329 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.268046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36193 (local address 127.0.97.1:36177)
0927 05:14:56.268985 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 05:14:56.268998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:14:56.269133 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:14:56.269468 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:14:56.269472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:14:56.269498 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:14:56.270214 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.270223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.271232 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.271238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.272527 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:14:56.272538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.273635 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.273641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:14:56.273847 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.273894 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 05:14:56.273979 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 05:14:56.274099 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":547,"threads_started":1}
I20260927 05:14:56.274418 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.268163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36177 (local address 127.0.97.2:36193)
0927 05:14:56.268387 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.268396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.269088 (+ 692us) server_negotiation.cc:407] Connection header received
0927 05:14:56.269281 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.269287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.269351 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.269444 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:14:56.270366 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.271096 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.272681 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.273423 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.273559 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.273655 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.274030 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.274161 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:14:56.274844 12151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3fc4fdbd6e4428b47dafe8bfcb3be7" candidate_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" is_pre_election: true
I20260927 05:14:56.275403 12151 raft_consensus.cc:2454] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be73b83c4dc4b81ba9b04f5a2d47d47 in term 0.
I20260927 05:14:56.275421 12017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3fc4fdbd6e4428b47dafe8bfcb3be7" candidate_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8d20394fe3419786f361d0496e1039" is_pre_election: true
I20260927 05:14:56.275933 12017 raft_consensus.cc:2454] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be73b83c4dc4b81ba9b04f5a2d47d47 in term 0.
I20260927 05:14:56.276072 11808 leader_election.cc:302] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be73b83c4dc4b81ba9b04f5a2d47d47, a4da1f3d82ff48bd893e92c0059f8a55; no voters:
I20260927 05:14:56.276490 12222 raft_consensus.cc:2781] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:14:56.276609 12222 raft_consensus.cc:482] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:14:56.276742 12222 raft_consensus.cc:3037] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:56.278920 12222 raft_consensus.cc:504] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.279565 12222 leader_election.cc:288] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [CANDIDATE]: Term 1 election: Requested vote from peers af8d20394fe3419786f361d0496e1039 (127.0.97.2:36193), a4da1f3d82ff48bd893e92c0059f8a55 (127.0.97.3:45935)
I20260927 05:14:56.279881 12017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3fc4fdbd6e4428b47dafe8bfcb3be7" candidate_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8d20394fe3419786f361d0496e1039"
I20260927 05:14:56.280108 12017 raft_consensus.cc:3037] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:56.280064 12151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3fc4fdbd6e4428b47dafe8bfcb3be7" candidate_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4da1f3d82ff48bd893e92c0059f8a55"
I20260927 05:14:56.280280 12151 raft_consensus.cc:3037] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:14:56.282588 12017 raft_consensus.cc:2454] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be73b83c4dc4b81ba9b04f5a2d47d47 in term 1.
I20260927 05:14:56.282828 12151 raft_consensus.cc:2454] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be73b83c4dc4b81ba9b04f5a2d47d47 in term 1.
I20260927 05:14:56.282976 11807 leader_election.cc:302] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be73b83c4dc4b81ba9b04f5a2d47d47, af8d20394fe3419786f361d0496e1039; no voters:
I20260927 05:14:56.283288 12222 raft_consensus.cc:2781] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:14:56.283712 12222 raft_consensus.cc:686] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 1 LEADER]: Becoming Leader. State: Replica: 6be73b83c4dc4b81ba9b04f5a2d47d47, State: Running, Role: LEADER
I20260927 05:14:56.284133 12222 consensus_queue.cc:237] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } }
I20260927 05:14:56.288703 11705 catalog_manager.cc:5495] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6be73b83c4dc4b81ba9b04f5a2d47d47 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be73b83c4dc4b81ba9b04f5a2d47d47" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 } health_report { overall_health: UNKNOWN } } }
W20260927 05:14:56.302533 12197 tablet.cc:2369] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:14:56.336805 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.331768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.0.97.1:44391)
0927 05:14:56.331980 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.331984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.332001 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:14:56.332174 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.332178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.332252 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.332362 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:14:56.332685 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.333401 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.334732 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.335219 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.335319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.335756 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:14:56.336290 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:14:56.336395 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.336461 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.336520 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:14:56.344838 12151 raft_consensus.cc:1261] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 1 FOLLOWER]: Refusing update from remote peer 6be73b83c4dc4b81ba9b04f5a2d47d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:14:56.344838 12017 raft_consensus.cc:1261] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 1 FOLLOWER]: Refusing update from remote peer 6be73b83c4dc4b81ba9b04f5a2d47d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:14:56.345821 12226 consensus_queue.cc:1035] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8d20394fe3419786f361d0496e1039" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:14:56.346148 12222 consensus_queue.cc:1035] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4da1f3d82ff48bd893e92c0059f8a55" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:14:56.357364 12229 mvcc.cc:205] Tried to move back new op lower bound from 7333831050618896384 to 7333831050383355904. Current Snapshot: MvccSnapshot[applied={T|T < 7333831050618896384}]
I20260927 05:14:56.372051 11705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:14:56.372274 11705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:14:56.383312 11705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60ac3e1a574441c9a99b0f9adb272602: Sending DeleteTablet for 3 replicas of tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7
I20260927 05:14:56.384307 11997 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38758
I20260927 05:14:56.384395 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58282
I20260927 05:14:56.385082 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:14:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46290
I20260927 05:14:56.386459 12241 tablet_replica.cc:330] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: stopping tablet replica
I20260927 05:14:56.386891 12241 raft_consensus.cc:2229] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:14:56.387678 12241 raft_consensus.cc:2258] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:56.389225 12240 tablet_replica.cc:330] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: stopping tablet replica
I20260927 05:14:56.389614 12240 raft_consensus.cc:2229] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:14:56.390081 12240 raft_consensus.cc:2258] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:56.390456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 11786
I20260927 05:14:56.390831 12239 tablet_replica.cc:330] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: stopping tablet replica
I20260927 05:14:56.391222 12239 raft_consensus.cc:2229] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:14:56.391659 12239 raft_consensus.cc:2258] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:14:56.392518 12240 ts_tablet_manager.cc:1905] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:14:56.393038 12241 ts_tablet_manager.cc:1905] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:14:56.396301 12239 ts_tablet_manager.cc:1905] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:14:56.399497 12240 ts_tablet_manager.cc:1918] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:56.399715 12240 log.cc:1196] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/fb3fc4fdbd6e4428b47dafe8bfcb3be7
I20260927 05:14:56.400292 12240 ts_tablet_manager.cc:1939] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P a4da1f3d82ff48bd893e92c0059f8a55: Deleting consensus metadata
I20260927 05:14:56.402030 11690 catalog_manager.cc:4841] TS a4da1f3d82ff48bd893e92c0059f8a55 (127.0.97.3:45935): tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (table test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]) successfully deleted
Sep 27 05:14:56 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:14:56.402561 12239 ts_tablet_manager.cc:1918] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:56.402714 12239 log.cc:1196] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/fb3fc4fdbd6e4428b47dafe8bfcb3be7
I20260927 05:14:56.403219 12239 ts_tablet_manager.cc:1939] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P af8d20394fe3419786f361d0496e1039: Deleting consensus metadata
I20260927 05:14:56.404884 11689 catalog_manager.cc:4841] TS af8d20394fe3419786f361d0496e1039 (127.0.97.2:36193): tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (table test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]) successfully deleted
I20260927 05:14:56.409849 12241 ts_tablet_manager.cc:1918] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:14:56.410029 12241 log.cc:1196] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/fb3fc4fdbd6e4428b47dafe8bfcb3be7
I20260927 05:14:56.410595 12241 ts_tablet_manager.cc:1939] T fb3fc4fdbd6e4428b47dafe8bfcb3be7 P 6be73b83c4dc4b81ba9b04f5a2d47d47: Deleting consensus metadata
I20260927 05:14:56.411111 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.392055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.97.1:44391)
0927 05:14:56.392214 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.392222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.392241 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:14:56.392296 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.392301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.392376 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.392466 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:56.394210 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.395427 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.397433 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.398121 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.398221 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.403516 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:56.403529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:56.403544 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:56.403588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:56.407840 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.408671 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.408679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.408683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.408861 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.409252 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.409263 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.409267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.409777 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:56.410207 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.410414 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.410705 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 05:14:56.412297 11914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:56.412563 11692 catalog_manager.cc:4841] TS 6be73b83c4dc4b81ba9b04f5a2d47d47 (127.0.97.1:44391): tablet fb3fc4fdbd6e4428b47dafe8bfcb3be7 (table test-table [id=7baedb8a2c2840038d12bdd1f7fb2a1d]) successfully deleted
I20260927 05:14:56.508253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 11786
I20260927 05:14:56.523152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 11925
Sep 27 05:14:56 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:14:56.537374 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.523866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.0.97.2:36193)
0927 05:14:56.524042 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.524048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.524075 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:14:56.524199 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.524202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.524332 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.524422 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:56.525595 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.526291 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.527677 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.528054 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.528152 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.532176 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:56.532187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:56.532201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:56.532228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:56.535118 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.535672 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.535678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.535680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.535810 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.536112 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.536115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.536117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.536435 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:56.536626 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.536863 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.537007 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:14:56.537999 12057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:56.626977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 11925
I20260927 05:14:56.643172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 12068
Sep 27 05:14:56 dist-test-slave-1t6p krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:14:56.657821 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:14:56.643884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.97.3:45935)
0927 05:14:56.644131 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 05:14:56.644136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:14:56.644155 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:14:56.644286 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:14:56.644296 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:14:56.644358 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:14:56.644479 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:14:56.645779 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.646493 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.647887 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:14:56.648272 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:14:56.648400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:14:56.652532 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:14:56.652545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:14:56.652562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:14:56.652591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:14:56.655196 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.655846 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.655854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.655856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.655968 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:14:56.656328 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:14:56.656335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:14:56.656338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:14:56.656702 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:14:56.656989 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:14:56.657296 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 05:14:56.657500 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 05:14:56.658417 12191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:56.745456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 12068
I20260927 05:14:56.761567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 11673
I20260927 05:14:56.762877 11735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:14:56.872165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 11673
I20260927 05:14:56.896075 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:14:57.231302 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:14:57.231521 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:14:57.576568 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:14:57.576 UTC [11259] LOG: received smart shutdown request
2026-09-27 05:14:57.580 UTC [11259] LOG: background worker "logical replication launcher" (PID 11268) exited with exit code 1
2026-09-27 05:14:57.580 UTC [11263] LOG: shutting down
2026-09-27 05:14:57.738 UTC [11259] LOG: database system is shut down
2026-09-27T05:14:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48488 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:14:57 dist-test-slave-1t6p krb5kdc[12255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:14:57 dist-test-slave-1t6p krb5kdc[12255](info): set up 2 sockets
Sep 27 05:14:57 dist-test-slave-1t6p krb5kdc[12255](info): commencing operation
krb5kdc: starting...
W20260927 05:14:58.971359 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.009s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:14:59 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:14:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:14:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 05:14:59.274220 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7OtwtO/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/postgres -l logfile start
2026-09-27 05:15:01.709 UTC [12285] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:15:01.709 UTC [12285] LOG: listening on IPv4 address "127.0.97.20", port 44273
2026-09-27 05:15:01.712 UTC [12285] LOG: listening on Unix socket "/tmp/.s.PGSQL.44273"
2026-09-27 05:15:01.733 UTC [12288] LOG: database system was shut down at 2026-09-27 05:15:00 UTC
2026-09-27 05:15:01.736 UTC [12285] LOG: database system is ready to accept connections
I20260927 05:15:02.703080 388 mini_postgres.cc:96] Postgres bound to 44273
2026-09-27 05:15:02.719 UTC [12296] FATAL: database "slave" does not exist
127.0.97.20:44273 - accepting connections
I20260927 05:15:02.719731 388 mini_ranger.cc:161] Starting Ranger...
I20260927 05:15:02.742660 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:15:03.200131 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:15:03.200242 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin
2026-09-27 05:15:03,493 [I] DB FLAVOR :POSTGRES
2026-09-27 05:15:03,493 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:15:03,494 [I] Checking connection..
2026-09-27 05:15:03,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:15:03,742 [I] Checking connection passed.
2026-09-27 05:15:03,742 [I] --------- Verifying version history table ---------
2026-09-27 05:15:03,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:15:03,964 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:15:03,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:15:04,200 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:15:04,200 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:15:04,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:15:04.433 UTC [12360] WARNING: there is no transaction in progress
2026-09-27 05:15:04,439 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:15:04,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:15:04,693 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:15:04,693 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:15:04,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:15:04,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:15:05,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1t6p.c.gcp-upstream.internal','N') ;"
2026-09-27 05:15:05,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:15:05,649 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:15:05,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:15:05,911 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:15:05,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:15:06,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:15:06,398 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:15:06,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:15:07.324 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.339 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.355 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.370 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.384 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.401 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.454 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.460 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.474 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.484 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.494 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.507 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.516 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.523 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.529 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:07.535 UTC [12468] WARNING: there is no transaction in progress
2026-09-27 05:15:08,129 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:15:08,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:15:08,362 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:15:08,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:15:08,613 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:15:08,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:15:08,871 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:15:08,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:15:09,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:15:09,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1t6p.c.gcp-upstream.internal';"
2026-09-27 05:15:09,589 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:15:09,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:15:09,831 [I] DB_PATCHES have already been applied
I20260927 05:15:09.837038 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7OtwtO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:15:09.837127 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 05:15:09.851046 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34349
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:15:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:15:10.984856 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.265s user 0.002s sys 0.034s
Sep 27, 2026 5:15:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:15:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 5:15:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:15:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:15:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:15:31.030848 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:15:31 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486131, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:15:31.179767 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:15:31.198717 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:15:32.023201 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:15:32.023317 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms
I20260927 05:15:32.023361 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:44273
2026-09-27 05:15:32,371 [I] DB FLAVOR :POSTGRES
2026-09-27 05:15:32,371 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:15:32,371 [I] Checking connection
2026-09-27 05:15:32,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:15:32,605 [I] connection success
2026-09-27 05:15:32,605 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:15:32,605 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:15:32,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:15:32,826 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:15:32,826 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:15:32,826 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:15:32,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:15:33,071 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:15:33.549674 388 mini_ranger_kms.cc:310] Created kms service
I20260927 05:15:33.748749 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:15:33.793229 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:15:33 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486131, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:34.186278 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:15:34.186584 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task7OtwtO/build/asan/bin/postgresql.jar:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7OtwtO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:15:34.186651 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 05:15:34.204738 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12648
I20260927 05:15:34.204870 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:34347
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:15:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 05:15:36.465500 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.286s user 0.087s sys 0.268s
I20260927 05:15:36.465761 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:15:36.465931 388 mini_ranger_kms.cc:401] 127.0.97.20:34347/kms/v1/keys
Sep 27 05:15:36 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790486136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:15:37 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:15:37 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:15:39 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:15:39.616922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38821
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:39.939558 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:39.939850 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:39.939992 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:39.947258 12699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:39.947361 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:39.947434 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:39.947533 12699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:39.947613 12699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:39.957546 12699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:34347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38821
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:39.958784 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:39.960109 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:39.972783 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:39.973521 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:39.973646 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:39.974200 12699 server_base.cc:1034] running on GCE node
I20260927 05:15:39.975085 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:39.976759 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:39.978000 12699 hybrid_clock.cc:648] HybridClock initialized: now 1790486139977942 us; error 47 us; skew 500 ppm
I20260927 05:15:39.978394 12699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:39 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:39.987422 12699 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:39.988945 12699 webserver.cc:460] Webserver started at http://127.0.97.62:33883/ using document root <none> and password file <none>
I20260927 05:15:39.989579 12699 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:39.989717 12699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:39.990003 12699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:15:39 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:40.040644 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "dc1b0c7f26e745e39cba2fb59e1178ea"
format_stamp: "Formatted at 2026-09-27 05:15:40 on dist-test-slave-1t6p"
server_key: "6d55f5fe259a7cdaebb5120f310bdfe7"
server_key_iv: "71e89bdb82b91913443c764f8419021c"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:40.041477 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "dc1b0c7f26e745e39cba2fb59e1178ea"
format_stamp: "Formatted at 2026-09-27 05:15:40 on dist-test-slave-1t6p"
server_key: "6d55f5fe259a7cdaebb5120f310bdfe7"
server_key_iv: "71e89bdb82b91913443c764f8419021c"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:40.046697 12699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:15:40.064330 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:40.066176 12699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 05:15:40.066357 12699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "dc1b0c7f26e745e39cba2fb59e1178ea"
format_stamp: "Formatted at 2026-09-27 05:15:40 on dist-test-slave-1t6p"
server_key: "6d55f5fe259a7cdaebb5120f310bdfe7"
server_key_iv: "71e89bdb82b91913443c764f8419021c"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:40.066608 12699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:40.098963 12699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:40.103756 12699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:40.104107 12699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:40.123762 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38821 every 8 connection(s)
I20260927 05:15:40.123760 12699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38821
I20260927 05:15:40.125487 12699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:40.127274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 12699
I20260927 05:15:40.127494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:15:40.128361 388 external_mini_cluster.cc:1428] Setting key 477fdfd40fb056f0c19f38251b21f5cd
I20260927 05:15:40.130160 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:40.147974 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Bootstrap starting.
I20260927 05:15:40.151444 12766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:40.152542 12766 log.cc:824] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:40.155562 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: No bootstrap required, opened a new log
Sep 27 05:15:40 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:40.163852 12766 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:40.164256 12766 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:40.164361 12766 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1b0c7f26e745e39cba2fb59e1178ea, State: Initialized, Role: FOLLOWER
I20260927 05:15:40.165014 12766 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:40.165287 12766 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:15:40.165462 12766 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:15:40.165824 12766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:40.168740 12766 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:40.168967 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:40.130370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.97.62:38821)
0927 05:15:40.145126 (+ 14756us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:40.145145 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:40.145191 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:15:40.147577 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:40.147606 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:40.148073 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:40.148504 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:40.150054 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:40.152036 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:40.153785 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:40.154758 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:40.154926 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:40.159754 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:40.159764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:40.159785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:40.159837 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:40.163899 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:40.164738 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:40.164746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:40.164753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:40.164918 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:40.165446 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:40.165452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:40.165458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:40.166193 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:40.166510 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:40.167319 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 05:15:40.167710 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14512,"thread_start_us":16657,"threads_started":1}
I20260927 05:15:40.169337 12766 leader_election.cc:302] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1b0c7f26e745e39cba2fb59e1178ea; no voters:
I20260927 05:15:40.169994 12766 leader_election.cc:288] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:15:40.170104 12771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:40.173571 12771 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 LEADER]: Becoming Leader. State: Replica: dc1b0c7f26e745e39cba2fb59e1178ea, State: Running, Role: LEADER
I20260927 05:15:40.174160 12771 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:40.174528 12766 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:40.177066 12772 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260927 05:15:40.177534 12772 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:40.178126 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1b0c7f26e745e39cba2fb59e1178ea. Latest consensus state: current_term: 1 leader_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260927 05:15:40.178455 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:40.179031 12776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:40.196161 12776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:40.209731 12776 catalog_manager.cc:1347] Generated new cluster ID: 76839f6e277943e58b281ca3643bc552
I20260927 05:15:40.209853 12776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:40.212304 12766 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:15:40.231142 12776 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:15:40.233857 12776 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:40.251863 12776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Generated new TSK 0
I20260927 05:15:40.252750 12776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:15:41.072172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:15:41 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:15:41.492580 12811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:41.493024 12811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:41.493233 12811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:41.505124 12811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:41.505352 12811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:41.505594 12811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:15:41.521340 12811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:41.522966 12811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:41.524618 12811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:41.542272 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:41.542295 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:41.542892 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:41.544099 12811 server_base.cc:1034] running on GCE node
I20260927 05:15:41.544765 12811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:41.545751 12811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:41.546972 12811 hybrid_clock.cc:648] HybridClock initialized: now 1790486141546906 us; error 65 us; skew 500 ppm
I20260927 05:15:41.547312 12811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:41 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:41.551333 12811 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:15:41.562438 12811 webserver.cc:460] Webserver started at http://127.0.97.1:41875/ using document root <none> and password file <none>
I20260927 05:15:41.563369 12811 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:41.563514 12811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:41.563884 12811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:15:41 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:41.586385 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "d2bb9d9b68e44a84b1de3134b282bfa1"
format_stamp: "Formatted at 2026-09-27 05:15:41 on dist-test-slave-1t6p"
server_key: "db80a77c5e4b8089e334506ea0ff17e0"
server_key_iv: "6d61b25b0d8b5780b3974352e325b277"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:41.589792 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d2bb9d9b68e44a84b1de3134b282bfa1"
format_stamp: "Formatted at 2026-09-27 05:15:41 on dist-test-slave-1t6p"
server_key: "db80a77c5e4b8089e334506ea0ff17e0"
server_key_iv: "6d61b25b0d8b5780b3974352e325b277"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:41.603255 12811 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260927 05:15:41.639931 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:41.644531 12811 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.006s
I20260927 05:15:41.644793 12811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "d2bb9d9b68e44a84b1de3134b282bfa1"
format_stamp: "Formatted at 2026-09-27 05:15:41 on dist-test-slave-1t6p"
server_key: "db80a77c5e4b8089e334506ea0ff17e0"
server_key_iv: "6d61b25b0d8b5780b3974352e325b277"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:41.645133 12811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:41.676971 12811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:41.682137 12811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:41.682456 12811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:41.683629 12811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:41.685567 12811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:41.685724 12811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:41.685920 12811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:41.686048 12811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:15:41 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:41.945828 12811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38745
I20260927 05:15:41.945868 12943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38745 every 8 connection(s)
I20260927 05:15:41.947595 12811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:15:41.951728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 12811
I20260927 05:15:41.951936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:15:41.952698 388 external_mini_cluster.cc:1428] Setting key f1aa8d567461aaa3c91e7a448ad53dca
Sep 27 05:15:41 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:15:41.995816 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:41.950472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40241 (local address 127.0.97.62:38821)
0927 05:15:41.976945 (+ 26473us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:41.976951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:41.976978 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:15:41.977771 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:41.977777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:41.977859 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:41.977957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:41.979974 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:41.981047 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:41.983241 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:41.983821 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:41.983925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:41.988093 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:41.988100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:41.988104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:41.988149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:41.991516 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:41.992336 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:41.992341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:41.992344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:41.992480 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:41.993058 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:41.993063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:41.993065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:41.993388 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:41.993519 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:41.995204 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0927 05:15:41.995424 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26337,"thread_start_us":31224,"threads_started":1}
I20260927 05:15:41.997501 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:41.950970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:40241)
0927 05:15:41.974131 (+ 23161us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:41.974192 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:41.977329 (+ 3137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:41.978385 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:41.978397 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:41.979020 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:41.979789 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:41.979811 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:41.981215 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:41.981220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:41.983066 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:41.983079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:41.984199 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:41.984204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:41.984477 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:41.985263 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:41.985289 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:41.987862 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:41.991703 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:41.991711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:41.991729 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:41.992208 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:41.992614 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:41.992619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:41.992623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:41.992908 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:41.993524 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:41.993532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:41.993940 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 05:15:41.994280 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 05:15:41.994793 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21565,"thread_start_us":21677,"threads_started":1}
I20260927 05:15:41.999673 12944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:42.000008 12944 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:42.000839 12944 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:42.003408 12731 ts_manager.cc:194] Registered new tserver with Master: d2bb9d9b68e44a84b1de3134b282bfa1 (127.0.97.1:38745)
I20260927 05:15:42.005457 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:15:42.028663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:15:42 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 05:15:42.447293 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:42.447577 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:42.447743 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:42.454823 12950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:42.454947 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:42.455128 12950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:15:42.464280 12950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:42.465433 12950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:42.466632 12950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:42.479552 12964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:42.479667 12967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:42.479700 12965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:42.480605 12950 server_base.cc:1034] running on GCE node
I20260927 05:15:42.481208 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:42.482095 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:42.483294 12950 hybrid_clock.cc:648] HybridClock initialized: now 1790486142483255 us; error 33 us; skew 500 ppm
I20260927 05:15:42.483544 12950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:42 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:42.486539 12950 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:15:42.487952 12950 webserver.cc:460] Webserver started at http://127.0.97.2:35079/ using document root <none> and password file <none>
I20260927 05:15:42.488586 12950 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:42.488720 12950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:42.489019 12950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:15:42 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:42.504187 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "1a230dfcfa8f41809495b7b1e605e873"
format_stamp: "Formatted at 2026-09-27 05:15:42 on dist-test-slave-1t6p"
server_key: "506df356fa4d369cb1d8c14381ef6536"
server_key_iv: "09c8b5aeccaeb3b6165f8e83e7ea0c9b"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:42.505067 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1a230dfcfa8f41809495b7b1e605e873"
format_stamp: "Formatted at 2026-09-27 05:15:42 on dist-test-slave-1t6p"
server_key: "506df356fa4d369cb1d8c14381ef6536"
server_key_iv: "09c8b5aeccaeb3b6165f8e83e7ea0c9b"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:42.510524 12950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:15:42.527370 12974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:42.528800 12950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 05:15:42.528964 12950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "1a230dfcfa8f41809495b7b1e605e873"
format_stamp: "Formatted at 2026-09-27 05:15:42 on dist-test-slave-1t6p"
server_key: "506df356fa4d369cb1d8c14381ef6536"
server_key_iv: "09c8b5aeccaeb3b6165f8e83e7ea0c9b"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:42.529175 12950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:42.557322 12950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:42.561017 12950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:42.561316 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:42.562211 12950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:42.563614 12950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:42.563684 12950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:42.563850 12950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:42.563900 12950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:42.597494 12950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41571
I20260927 05:15:42.597538 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41571 every 8 connection(s)
I20260927 05:15:42.598793 12950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:15:42.608490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 12950
I20260927 05:15:42.608703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:15:42.609287 388 external_mini_cluster.cc:1428] Setting key 7a47d97cd0671cb69bf2eb69abc54f1c
Sep 27 05:15:42 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:42.620731 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:42.601423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55817 (local address 127.0.97.62:38821)
0927 05:15:42.601883 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:42.601889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:42.602874 (+ 985us) server_negotiation.cc:407] Connection header received
0927 05:15:42.604699 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:42.604703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:42.604778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:42.604873 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:42.606211 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:42.607038 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:42.609186 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:42.609602 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:42.609703 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:42.613580 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:42.613589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:42.613594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:42.613647 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:42.617605 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:42.618324 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:42.618333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:42.618338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:42.618495 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:42.618895 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:42.618902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:42.618908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:42.619314 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:42.619454 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:42.620334 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 05:15:42.620487 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260927 05:15:42.622295 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:42.601985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:55817)
0927 05:15:42.602710 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:42.602752 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:42.604453 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:42.605113 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:42.605122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:42.605513 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:42.606055 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:42.606071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:42.607198 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:42.607202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:42.609025 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:42.609043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:42.609825 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:42.609833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:42.610135 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:42.610954 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:42.610975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:42.613370 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:42.617748 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:42.617755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:42.617767 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:42.618203 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:42.618611 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:42.618615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:42.618617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:42.618785 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:42.619460 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:42.619469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:42.619902 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 05:15:42.620262 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 05:15:42.620834 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":298,"threads_started":1}
I20260927 05:15:42.623768 13087 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:42.624053 13087 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:42.624715 13087 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:42.626106 12731 ts_manager.cc:194] Registered new tserver with Master: 1a230dfcfa8f41809495b7b1e605e873 (127.0.97.2:41571)
I20260927 05:15:42.626876 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55817
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:15:42.662446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:42.975088 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:42.975414 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:42.975540 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:42.982738 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:42.982834 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:42.982947 13093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:15:42.992357 13093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:42.993543 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:42.994767 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:43.007516 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:43.007635 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:43.007737 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:43.008450 13093 server_base.cc:1034] running on GCE node
I20260927 05:15:43.009093 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:43.009305 12944 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260927 05:15:43.010147 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:43.011372 13093 hybrid_clock.cc:648] HybridClock initialized: now 1790486143011303 us; error 64 us; skew 500 ppm
I20260927 05:15:43.011650 13093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:43.014531 13093 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:15:43.015947 13093 webserver.cc:460] Webserver started at http://127.0.97.3:38183/ using document root <none> and password file <none>
I20260927 05:15:43.016610 13093 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:43.016745 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:43.017071 13093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:43.030486 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "d791fd9476644099abd8fc0dde876b67"
format_stamp: "Formatted at 2026-09-27 05:15:43 on dist-test-slave-1t6p"
server_key: "cb6de0cae32eafe30ad8efe1395fe963"
server_key_iv: "9ea3d44b1b88888ea8e59e1671dcae08"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:43.031409 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d791fd9476644099abd8fc0dde876b67"
format_stamp: "Formatted at 2026-09-27 05:15:43 on dist-test-slave-1t6p"
server_key: "cb6de0cae32eafe30ad8efe1395fe963"
server_key_iv: "9ea3d44b1b88888ea8e59e1671dcae08"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:43.036844 13093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:15:43.051057 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:43.052572 13093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:15:43.052740 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "d791fd9476644099abd8fc0dde876b67"
format_stamp: "Formatted at 2026-09-27 05:15:43 on dist-test-slave-1t6p"
server_key: "cb6de0cae32eafe30ad8efe1395fe963"
server_key_iv: "9ea3d44b1b88888ea8e59e1671dcae08"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:43.052954 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:43.078147 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:43.081640 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:43.081929 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:43.082752 13093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:43.083990 13093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:43.084059 13093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:43.084149 13093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:43.084190 13093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:43.116065 13093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33391
I20260927 05:15:43.116106 13220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33391 every 8 connection(s)
I20260927 05:15:43.117269 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:15:43.122676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13093
I20260927 05:15:43.122881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:15:43.123515 388 external_mini_cluster.cc:1428] Setting key e147cae0c90485c920f2c5cb1375c349
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:43.136708 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:43.119587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48907 (local address 127.0.97.62:38821)
0927 05:15:43.119758 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:43.119762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:43.120843 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 05:15:43.122627 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:43.122632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:43.122707 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:43.122831 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:43.124651 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.125812 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.127545 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.127988 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.128129 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:43.131478 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:43.131484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:43.131489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:43.131527 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:43.134190 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.134815 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.134818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.134821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.134949 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.135422 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.135426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.135427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.135668 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:43.135784 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:43.136341 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 05:15:43.136488 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:15:43.137964 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:43.120039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:48907)
0927 05:15:43.120708 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:43.120742 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:43.122324 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:43.123116 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:43.123127 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:43.123711 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:43.124463 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:43.124493 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.125960 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.125971 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:43.127397 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:43.127409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.128203 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.128211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:43.128423 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:43.129071 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:43.129093 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:43.131272 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:43.134306 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:43.134312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:43.134322 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:43.134712 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:43.135080 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:43.135088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:43.135094 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:43.135320 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:43.135778 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:43.135783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:43.136038 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 05:15:43.136250 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 05:15:43.136567 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":292,"threads_started":1}
I20260927 05:15:43.139283 13221 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:43.139567 13221 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:43.140226 13221 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:43.141445 12731 ts_manager.cc:194] Registered new tserver with Master: d791fd9476644099abd8fc0dde876b67 (127.0.97.3:33391)
I20260927 05:15:43.142138 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48907
I20260927 05:15:43.150882 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:15:43.152559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 12811
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:15:43.168411 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:43.153344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.0.97.1:38745)
0927 05:15:43.153925 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:43.153933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:43.153962 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:15:43.154048 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:43.154055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:43.154232 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:43.154391 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:43.155892 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.156989 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.158500 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.159046 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.159188 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:43.163256 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:43.163268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:43.163280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:43.163317 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:43.165991 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.166601 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.166607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.166610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.166748 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.167114 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.167118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.167120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.167435 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:43.167669 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:43.167926 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 05:15:43.168134 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260927 05:15:43.169624 12939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:43.254042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 12811
I20260927 05:15:43.270298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 12950
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:15:43.286024 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:43.271203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.0.97.2:41571)
0927 05:15:43.271727 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:43.271734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:43.271758 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:15:43.271816 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:43.271822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:43.272121 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:43.272305 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:43.273663 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.274611 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.276091 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.276639 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.276744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:43.280822 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:43.280832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:43.280845 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:43.280874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:43.283609 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.284277 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.284284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.284286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.284401 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.284784 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.284795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.284797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.285089 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:43.285318 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:43.285561 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 05:15:43.285750 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":262,"threads_started":1}
I20260927 05:15:43.287346 13082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:43.370896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 12950
I20260927 05:15:43.390720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13093
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:15:43.405802 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:43.391678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.97.3:33391)
0927 05:15:43.392275 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:43.392282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:43.392309 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:15:43.392372 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:43.392378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:43.392578 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:43.392705 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:43.393923 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.394892 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.396219 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:43.396740 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:43.396832 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:43.400844 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:43.400853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:43.400867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:43.400901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:43.403548 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.404095 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.404102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.404104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.404236 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:43.404534 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:43.404537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:43.404538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:43.404820 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:43.405034 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:43.405236 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 05:15:43.405410 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":293,"threads_started":1}
I20260927 05:15:43.407115 13216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:43.488176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13093
I20260927 05:15:43.503605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 12699
I20260927 05:15:43.504858 12761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:43.612025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 12699
I20260927 05:15:43.628700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--webserver_interface=127.0.97.62
--webserver_port=33883
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38821
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:43.941730 13232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:43.942051 13232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:43.942221 13232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:43.949329 13232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:43.949467 13232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:43.949606 13232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:43.949708 13232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:43.949800 13232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:43.959440 13232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38821
--ranger_config_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:43.960650 13232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:43.961969 13232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:43.973529 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:43.973588 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:43.973552 13240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:43.974505 13232 server_base.cc:1034] running on GCE node
I20260927 05:15:43.975306 13232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:43.976652 13232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:43.977837 13232 hybrid_clock.cc:648] HybridClock initialized: now 1790486143977798 us; error 34 us; skew 500 ppm
I20260927 05:15:43.978199 13232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:43 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:43.981302 13232 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:43.982604 13232 webserver.cc:460] Webserver started at http://127.0.97.62:33883/ using document root <none> and password file <none>
I20260927 05:15:43.983258 13232 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:43.983364 13232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:15:44 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:44.048710 13232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:15:44.051600 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:44.052934 13232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:44.053125 13232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "dc1b0c7f26e745e39cba2fb59e1178ea"
format_stamp: "Formatted at 2026-09-27 05:15:40 on dist-test-slave-1t6p"
server_key: "6d55f5fe259a7cdaebb5120f310bdfe7"
server_key_iv: "71e89bdb82b91913443c764f8419021c"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:44.054057 13232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:44.071393 13232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:44.074721 13232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:44.074993 13232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:44.092658 13232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38821
I20260927 05:15:44.092682 13299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38821 every 8 connection(s)
I20260927 05:15:44.094030 13232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:44.098788 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13232
I20260927 05:15:44.099650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38745
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=41875
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:15:44.100512 13300 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:15:44.103593 13300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Bootstrap starting.
I20260927 05:15:44.121595 13300 log.cc:824] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:44.129866 13300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:15:44.130376 13300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Bootstrap complete.
I20260927 05:15:44.138965 13300 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:44.139892 13300 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1b0c7f26e745e39cba2fb59e1178ea, State: Initialized, Role: FOLLOWER
I20260927 05:15:44.140436 13300 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:44.140673 13300 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:15:44.140801 13300 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:15:44.140990 13300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:15:44.144616 13300 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:44.145080 13300 leader_election.cc:302] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1b0c7f26e745e39cba2fb59e1178ea; no voters:
I20260927 05:15:44.145728 13300 leader_election.cc:288] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:15:44.145933 13304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:15:44.146983 13304 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [term 2 LEADER]: Becoming Leader. State: Replica: dc1b0c7f26e745e39cba2fb59e1178ea, State: Running, Role: LEADER
I20260927 05:15:44.147644 13300 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:44.147696 13304 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260927 05:15:44.150493 13306 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1b0c7f26e745e39cba2fb59e1178ea. Latest consensus state: current_term: 2 leader_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260927 05:15:44.150938 13306 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:44.151568 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b0c7f26e745e39cba2fb59e1178ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260927 05:15:44.151903 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:44.160060 13313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:44.165934 13313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:44.166683 13300 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:15:44.167364 13313 catalog_manager.cc:1259] Loaded cluster ID: 76839f6e277943e58b281ca3643bc552
I20260927 05:15:44.167477 13313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:44.171468 13313 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:44.172927 13313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Loaded TSK: 0
I20260927 05:15:44.174373 13313 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:15:44.545540 13302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:44.545995 13302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:44.546162 13302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:44.557561 13302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:44.557772 13302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:44.558040 13302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:15:44.573485 13302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41875
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:44.575238 13302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:44.577033 13302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:44.603402 13337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:44.607337 13338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:44.610592 13302 server_base.cc:1034] running on GCE node
W20260927 05:15:44.610769 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:44.611517 13302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:44.612496 13302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:44.613718 13302 hybrid_clock.cc:648] HybridClock initialized: now 1790486144613663 us; error 49 us; skew 500 ppm
I20260927 05:15:44.614077 13302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:44 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:44.617851 13302 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:15:44.619746 13302 webserver.cc:460] Webserver started at http://127.0.97.1:41875/ using document root <none> and password file <none>
I20260927 05:15:44.620451 13302 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:44.620584 13302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:15:44 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:44.639571 13302 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:15:44.642385 13348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:44.644160 13302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:44.644330 13302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "d2bb9d9b68e44a84b1de3134b282bfa1"
format_stamp: "Formatted at 2026-09-27 05:15:41 on dist-test-slave-1t6p"
server_key: "db80a77c5e4b8089e334506ea0ff17e0"
server_key_iv: "6d61b25b0d8b5780b3974352e325b277"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:44.645665 13302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:44.689620 13302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:44.694913 13302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:44.695226 13302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:44.696426 13302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:44.698267 13302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:44.698379 13302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:44.698537 13302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:44.698649 13302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:44.760462 13302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38745
I20260927 05:15:44.760512 13461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38745 every 8 connection(s)
I20260927 05:15:44.762234 13302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:15:44.772009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13302
I20260927 05:15:44.772718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41571
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=35079
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:15:44 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:44.786783 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:44.765676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:34829)
0927 05:15:44.766481 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:44.766521 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:44.768522 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:44.770022 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:44.770033 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:44.770567 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:44.771177 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:44.771201 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:44.772803 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:44.772807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:44.773994 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:44.774006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:44.774986 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:44.774994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:44.775301 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:44.775931 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:44.775951 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:44.777939 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:44.781783 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:44.781792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:44.781805 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:44.782381 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:44.782847 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:44.782853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:44.782856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:44.783112 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:44.784034 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:44.784044 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:44.784417 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 05:15:44.784666 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:15:44.785217 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":966,"threads_started":1}
I20260927 05:15:44.788694 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:44.766559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34829 (local address 127.0.97.62:38821)
0927 05:15:44.767545 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:44.767563 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:44.767611 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:15:44.769139 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:44.769158 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:44.769539 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:44.769877 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:44.771379 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:44.772633 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:44.774137 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:44.774743 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:44.774965 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:44.778177 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:44.778184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:44.778197 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:44.778236 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:44.781640 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:44.782530 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:44.782536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:44.782541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:44.782734 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:44.783256 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:44.783264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:44.783270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:44.783888 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:44.785686 (+ 1798us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:44.786902 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0927 05:15:44.787353 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":540,"threads_started":1}
I20260927 05:15:44.792362 13462 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:44.792702 13462 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:44.793586 13462 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
W20260927 05:15:44.796909 13462 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38821 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:15:44.797142 13462 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:44.797401 13462 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:44.798718 13462 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:44.799060 13462 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
W20260927 05:15:44.800340 13462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:15:45.185912 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:45.186329 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:45.186494 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:45.197778 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:45.197918 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:45.198160 13466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:15:45.213624 13466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35079
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:45.215384 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:45.217085 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:45.233023 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:45.233053 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:45.233088 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:45.235066 13466 server_base.cc:1034] running on GCE node
I20260927 05:15:45.235780 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:45.236927 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:45.238246 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790486145238135 us; error 95 us; skew 500 ppm
I20260927 05:15:45.238606 13466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:45.246546 13466 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:15:45.248404 13466 webserver.cc:460] Webserver started at http://127.0.97.2:35079/ using document root <none> and password file <none>
I20260927 05:15:45.249289 13466 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:45.249435 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:45.271054 13466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:15:45.274338 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:45.275791 13466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:45.275975 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "1a230dfcfa8f41809495b7b1e605e873"
format_stamp: "Formatted at 2026-09-27 05:15:42 on dist-test-slave-1t6p"
server_key: "506df356fa4d369cb1d8c14381ef6536"
server_key_iv: "09c8b5aeccaeb3b6165f8e83e7ea0c9b"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:45.276924 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:45.321703 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:45.326944 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:45.327322 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:45.335887 13466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:45.339833 13466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:45.339948 13466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:45.340065 13466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:45.340138 13466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:45.389513 13466 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41571
I20260927 05:15:45.389598 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41571 every 8 connection(s)
I20260927 05:15:45.391410 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:15:45.393292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13466
I20260927 05:15:45.393967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33391
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=38183
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:45.421586 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:45.399114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41369 (local address 127.0.97.62:38821)
0927 05:15:45.403389 (+ 4275us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:45.403396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:45.403417 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:15:45.403482 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:45.403488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:45.403571 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:45.403682 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:45.405816 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:45.406993 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:45.409114 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:45.409641 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:45.409727 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:45.413796 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:45.413802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:45.413806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:45.413841 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:45.417540 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:45.418527 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:45.418535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:45.418540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:45.418712 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:45.419300 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:45.419308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:45.419315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:45.419695 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:45.419828 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:45.420832 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 05:15:45.421095 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4145,"thread_start_us":5382,"threads_started":1}
I20260927 05:15:45.424264 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:45.398273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:41369)
0927 05:15:45.399444 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:45.399491 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:45.401870 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:45.404149 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:45.404165 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:45.404760 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:45.405595 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:45.405621 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:45.407232 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:45.407236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:45.408946 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:45.408958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:45.410029 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:45.410035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:45.410290 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:45.410985 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:45.411041 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:45.413577 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:45.417714 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:45.417725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:45.417743 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:45.418379 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:45.418890 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:45.418897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:45.418903 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:45.419162 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:45.419940 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:45.419951 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:45.420388 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 05:15:45.422160 (+ 1772us) client_negotiation.cc:240] Negotiation successful
0927 05:15:45.422754 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":814,"threads_started":1}
I20260927 05:15:45.426481 13606 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:45.426807 13606 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:45.427652 13606 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:45.429602 13265 ts_manager.cc:194] Registered new tserver with Master: 1a230dfcfa8f41809495b7b1e605e873 (127.0.97.2:41571)
I20260927 05:15:45.431800 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41369
I20260927 05:15:45.800477 13462 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:45.800885 13462 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:45.802270 13265 ts_manager.cc:194] Registered new tserver with Master: d2bb9d9b68e44a84b1de3134b282bfa1 (127.0.97.1:38745)
I20260927 05:15:45.803563 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34829
W20260927 05:15:45.831408 13610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:45.831707 13610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:45.831871 13610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:45.839377 13610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:45.839509 13610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:45.839697 13610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:15:45.852000 13610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38183
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:45.853539 13610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:45.855173 13610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:45.883642 13617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:45.884629 13616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:45.899468 13610 server_base.cc:1034] running on GCE node
W20260927 05:15:45.904037 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:45.907465 13610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:45.909785 13610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:45.911176 13610 hybrid_clock.cc:648] HybridClock initialized: now 1790486145911068 us; error 56 us; skew 500 ppm
I20260927 05:15:45.911695 13610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:45.916152 13610 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:15:45.932251 13610 webserver.cc:460] Webserver started at http://127.0.97.3:38183/ using document root <none> and password file <none>
I20260927 05:15:45.933269 13610 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:45.933485 13610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:15:45 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:45.967617 13610 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:15:45.996229 13627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:45.998450 13610 fs_manager.cc:725] Time spent opening block manager: real 0.029s user 0.007s sys 0.012s
I20260927 05:15:45.998651 13610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "d791fd9476644099abd8fc0dde876b67"
format_stamp: "Formatted at 2026-09-27 05:15:43 on dist-test-slave-1t6p"
server_key: "cb6de0cae32eafe30ad8efe1395fe963"
server_key_iv: "9ea3d44b1b88888ea8e59e1671dcae08"
server_key_version: "kuduclusterkey@0"
I20260927 05:15:46.000182 13610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:46.027479 13610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:46.031673 13610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:46.031994 13610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:46.033001 13610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:46.034497 13610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:46.034616 13610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:46.034752 13610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:46.034839 13610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:46.075625 13610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33391
I20260927 05:15:46.075721 13739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33391 every 8 connection(s)
I20260927 05:15:46.077373 13610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:15:46.078871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13610
Sep 27 05:15:46 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:46.129873 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.080044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.97.62:38821)
0927 05:15:46.107737 (+ 27693us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.107746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.110732 (+ 2986us) server_negotiation.cc:407] Connection header received
0927 05:15:46.110800 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.110806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.110917 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.111058 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:46.113701 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.114795 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.116700 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.117520 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.117628 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.121341 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:46.121347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:46.121351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:46.121404 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:46.125906 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.127083 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.127093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.127097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.127270 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.128051 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.128057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.128061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.128417 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:46.128670 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.129263 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.129477 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":27538,"thread_start_us":28981,"threads_started":1}
I20260927 05:15:46.133006 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.080324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58231 (local address 127.0.97.62:38821)
0927 05:15:46.109014 (+ 28690us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.109023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.112180 (+ 3157us) server_negotiation.cc:407] Connection header received
0927 05:15:46.114375 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.114380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.114462 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.114531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:46.116676 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.117976 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.120540 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.121224 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.121305 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.125615 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:46.125627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:46.125633 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:46.125678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:46.129137 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.129996 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.130004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.130009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.130127 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.130720 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.130726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.130730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.131079 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:46.131192 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.132420 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.132623 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":28557,"thread_start_us":27160,"threads_started":1}
I20260927 05:15:46.134946 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.080709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:58231)
0927 05:15:46.109797 (+ 29088us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.109869 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.114088 (+ 4219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.115105 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.115119 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.115725 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:46.116469 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.116492 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.118163 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.118168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.120371 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.120387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.121670 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.121679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.121969 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.122748 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:46.122777 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:46.125315 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:46.129281 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:46.129289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:46.129306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:46.129852 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:46.130353 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:46.130357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:46.130359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:46.130570 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:46.131555 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:46.131561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:46.131983 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.133467 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.133991 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28679,"thread_start_us":28440,"threads_started":1}
I20260927 05:15:46.136327 13740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260927 05:15:46.136627 13740 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:46.137295 13740 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260927 05:15:46.138648 13265 ts_manager.cc:194] Registered new tserver with Master: d791fd9476644099abd8fc0dde876b67 (127.0.97.3:33391)
I20260927 05:15:46.139505 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58231
I20260927 05:15:46.145799 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:15:46.171964 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.148873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.0.97.62:38821)
0927 05:15:46.149044 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.149053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.155935 (+ 6882us) server_negotiation.cc:407] Connection header received
0927 05:15:46.156117 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.156123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.156217 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.156328 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:46.158353 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.159483 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.161507 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.162165 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.162304 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.165607 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:46.165614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:46.165619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:46.165655 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:46.169061 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.169843 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.169848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.169852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.170006 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.170409 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.170414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.170417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.170763 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:46.171040 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.171360 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.171586 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:15:46.177737 13265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:15:46.189525 13265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:15:46.246865 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.230061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41571 (local address 127.0.0.1:42704)
0927 05:15:46.235923 (+ 5862us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.235940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.236050 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.238022 (+ 1972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.238030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.238058 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:46.238419 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.238439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.240278 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.240283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.242375 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.242388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.245843 (+ 3455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.245856 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.246120 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.246191 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.246326 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.246514 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5749,"thread_start_us":6071,"threads_started":1}
I20260927 05:15:46.248594 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.224007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.97.2:41571)
0927 05:15:46.237012 (+ 13005us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.237021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.237049 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:15:46.237159 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.237166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.237440 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.237627 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:46.238614 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.240103 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.242542 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.244001 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.244120 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.244321 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.248169 (+ 3848us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.248322 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12848,"thread_start_us":13776,"threads_started":1}
I20260927 05:15:46.255340 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.212085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38745 (local address 127.0.0.1:52828)
0927 05:15:46.212893 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.212952 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.213121 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.245599 (+ 32478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.245607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.245645 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:46.246188 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.246204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.249834 (+ 3630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.249841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.252074 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.252086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.254455 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.254462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.254628 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.254666 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.254740 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.254870 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":4,"thread_start_us":380,"threads_started":1}
I20260927 05:15:46.255259 13541 tablet_service.cc:1467] Processing CreateTablet for tablet 133b9e2d9816475999f6f588a49857bc (DEFAULT_TABLE table=test-table [id=79903cf183934d9a99dca217f57a4fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:46.256630 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.212765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.97.1:38745)
0927 05:15:46.244935 (+ 32170us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.244944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.244981 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:15:46.245074 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.245081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.245335 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.245576 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:46.247358 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.249681 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.252696 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.254082 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.255873 (+ 1791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.256093 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.256171 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.256327 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31999,"thread_start_us":31866,"threads_started":1}
I20260927 05:15:46.259181 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.213758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33391 (local address 127.0.0.1:42572)
0927 05:15:46.223238 (+ 9480us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.223257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.223375 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.252314 (+ 28939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.252318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.252342 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:46.252653 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.252663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.254606 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.254611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.256579 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.256593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.258446 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.258453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.258616 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.258649 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.258721 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.258845 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9347,"thread_start_us":9702,"threads_started":1}
I20260927 05:15:46.260331 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.214472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.97.3:33391)
0927 05:15:46.251599 (+ 37127us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.251606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.251634 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:15:46.251726 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.251737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.252030 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.252260 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:46.252866 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.254428 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.256781 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.258200 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.259559 (+ 1359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.259775 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.259843 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.259995 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36953,"thread_start_us":36994,"threads_started":1}
I20260927 05:15:46.263876 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b9e2d9816475999f6f588a49857bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:46.264120 13675 tablet_service.cc:1467] Processing CreateTablet for tablet 133b9e2d9816475999f6f588a49857bc (DEFAULT_TABLE table=test-table [id=79903cf183934d9a99dca217f57a4fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:46.265600 13675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b9e2d9816475999f6f588a49857bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:46.275130 13397 tablet_service.cc:1467] Processing CreateTablet for tablet 133b9e2d9816475999f6f588a49857bc (DEFAULT_TABLE table=test-table [id=79903cf183934d9a99dca217f57a4fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:46.279072 13397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b9e2d9816475999f6f588a49857bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:46.284121 13763 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Bootstrap starting.
I20260927 05:15:46.288442 13763 tablet_bootstrap.cc:654] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Neither blocks nor log segments found. Creating new log.
Sep 27 05:15:46 dist-test-slave-1t6p krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:15:46.294775 13763 log.cc:824] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:46.298277 13762 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Bootstrap starting.
I20260927 05:15:46.301342 13763 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: No bootstrap required, opened a new log
I20260927 05:15:46.301474 13765 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Bootstrap starting.
I20260927 05:15:46.301616 13763 ts_tablet_manager.cc:1397] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260927 05:15:46.304785 13765 tablet_bootstrap.cc:654] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:46.307124 13765 log.cc:824] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:46.311923 13763 raft_consensus.cc:348] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.312381 13763 raft_consensus.cc:374] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:46.312507 13763 raft_consensus.cc:729] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a230dfcfa8f41809495b7b1e605e873, State: Initialized, Role: FOLLOWER
I20260927 05:15:46.313258 13763 consensus_queue.cc:260] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.330147 13762 tablet_bootstrap.cc:654] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:46.331972 13762 log.cc:824] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:46.361738 13762 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: No bootstrap required, opened a new log
I20260927 05:15:46.362239 13762 ts_tablet_manager.cc:1397] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.003s
I20260927 05:15:46.373988 13765 tablet_bootstrap.cc:492] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: No bootstrap required, opened a new log
I20260927 05:15:46.377483 13762 raft_consensus.cc:348] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.378602 13762 raft_consensus.cc:374] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:46.378701 13606 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260927 05:15:46.378770 13762 raft_consensus.cc:729] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d791fd9476644099abd8fc0dde876b67, State: Initialized, Role: FOLLOWER
I20260927 05:15:46.379547 13762 consensus_queue.cc:260] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.382650 13765 ts_tablet_manager.cc:1397] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Time spent bootstrapping tablet: real 0.081s user 0.024s sys 0.044s
I20260927 05:15:46.384500 13763 ts_tablet_manager.cc:1428] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Time spent starting tablet: real 0.083s user 0.024s sys 0.052s
I20260927 05:15:46.387461 13762 ts_tablet_manager.cc:1428] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Time spent starting tablet: real 0.025s user 0.011s sys 0.007s
I20260927 05:15:46.387846 13740 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260927 05:15:46.395051 13765 raft_consensus.cc:348] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.395525 13765 raft_consensus.cc:374] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:46.395639 13765 raft_consensus.cc:729] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bb9d9b68e44a84b1de3134b282bfa1, State: Initialized, Role: FOLLOWER
I20260927 05:15:46.396324 13765 consensus_queue.cc:260] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
W20260927 05:15:46.399451 13607 tablet.cc:2369] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:46.408329 13765 ts_tablet_manager.cc:1428] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Time spent starting tablet: real 0.026s user 0.011s sys 0.011s
I20260927 05:15:46.408658 13462 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
W20260927 05:15:46.516774 13463 tablet.cc:2369] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:46.559114 13767 raft_consensus.cc:482] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:15:46.559478 13767 raft_consensus.cc:504] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.561798 13767 leader_election.cc:288] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bb9d9b68e44a84b1de3134b282bfa1 (127.0.97.1:38745), d791fd9476644099abd8fc0dde876b67 (127.0.97.3:33391)
I20260927 05:15:46.571130 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.562727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57795 (local address 127.0.97.1:38745)
0927 05:15:46.562915 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.562922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.563137 (+ 215us) server_negotiation.cc:407] Connection header received
0927 05:15:46.563249 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.563258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.563342 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.563432 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:46.564124 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.565479 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.568346 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.569808 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.569958 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.570085 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.570699 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.570861 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:15:46.572157 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.562305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38745 (local address 127.0.97.2:57795)
0927 05:15:46.562894 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.562914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.563086 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.563498 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.563503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.563529 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:46.563936 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.563956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.565754 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.565762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.568185 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.568198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.570251 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.570259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.570479 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.570521 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.571672 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.571815 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":1928,"threads_started":1}
I20260927 05:15:46.573293 13417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133b9e2d9816475999f6f588a49857bc" candidate_uuid: "1a230dfcfa8f41809495b7b1e605e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" is_pre_election: true
I20260927 05:15:46.573822 13417 raft_consensus.cc:2454] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a230dfcfa8f41809495b7b1e605e873 in term 0.
I20260927 05:15:46.574566 13495 leader_election.cc:302] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a230dfcfa8f41809495b7b1e605e873, d2bb9d9b68e44a84b1de3134b282bfa1; no voters:
I20260927 05:15:46.575129 13767 raft_consensus.cc:2781] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:15:46.575310 13767 raft_consensus.cc:482] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:15:46.575428 13767 raft_consensus.cc:3037] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:46.578670 13767 raft_consensus.cc:504] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.580113 13417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133b9e2d9816475999f6f588a49857bc" candidate_uuid: "1a230dfcfa8f41809495b7b1e605e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1"
I20260927 05:15:46.580382 13417 raft_consensus.cc:3037] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:46.585048 13417 raft_consensus.cc:2454] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a230dfcfa8f41809495b7b1e605e873 in term 1.
I20260927 05:15:46.585794 13495 leader_election.cc:302] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a230dfcfa8f41809495b7b1e605e873, d2bb9d9b68e44a84b1de3134b282bfa1; no voters:
I20260927 05:15:46.588028 13767 leader_election.cc:288] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [CANDIDATE]: Term 1 election: Requested vote from peers d2bb9d9b68e44a84b1de3134b282bfa1 (127.0.97.1:38745), d791fd9476644099abd8fc0dde876b67 (127.0.97.3:33391)
I20260927 05:15:46.588416 13767 raft_consensus.cc:2781] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:46.588791 13767 raft_consensus.cc:686] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 1 LEADER]: Becoming Leader. State: Replica: 1a230dfcfa8f41809495b7b1e605e873, State: Running, Role: LEADER
I20260927 05:15:46.589389 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.563980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47679 (local address 127.0.97.3:33391)
0927 05:15:46.564163 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.564168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.567559 (+ 3391us) server_negotiation.cc:407] Connection header received
0927 05:15:46.574490 (+ 6931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.574496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.574584 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.574651 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:46.575532 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.576589 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.585862 (+ 9273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.587445 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.587603 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.587706 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.589037 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.589170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:15:46.588887 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.564019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33391 (local address 127.0.97.2:47679)
0927 05:15:46.567450 (+ 3431us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:46.567469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:46.574306 (+ 6837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:46.574864 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:46.574868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:46.574895 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:46.575321 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.575337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.576787 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.576792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.578994 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:46.585693 (+ 6699us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.587882 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.587892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:46.588127 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.588177 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 05:15:46.588300 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 05:15:46.588463 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3306,"thread_start_us":3158,"threads_started":1}
I20260927 05:15:46.589555 13767 consensus_queue.cc:237] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 05:15:46.591171 13695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133b9e2d9816475999f6f588a49857bc" candidate_uuid: "1a230dfcfa8f41809495b7b1e605e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d791fd9476644099abd8fc0dde876b67" is_pre_election: true
I20260927 05:15:46.591701 13695 raft_consensus.cc:2454] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a230dfcfa8f41809495b7b1e605e873 in term 0.
I20260927 05:15:46.592175 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133b9e2d9816475999f6f588a49857bc" candidate_uuid: "1a230dfcfa8f41809495b7b1e605e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d791fd9476644099abd8fc0dde876b67"
I20260927 05:15:46.592398 13694 raft_consensus.cc:3037] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:46.597102 13694 raft_consensus.cc:2454] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a230dfcfa8f41809495b7b1e605e873 in term 1.
I20260927 05:15:46.601046 13265 catalog_manager.cc:5495] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a230dfcfa8f41809495b7b1e605e873 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1a230dfcfa8f41809495b7b1e605e873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a230dfcfa8f41809495b7b1e605e873" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } health_report { overall_health: HEALTHY } } }
W20260927 05:15:46.613881 13741 tablet.cc:2369] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:46.684834 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.679117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.0.97.2:41571)
0927 05:15:46.679348 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.679355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.679616 (+ 261us) server_negotiation.cc:407] Connection header received
0927 05:15:46.679823 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.679826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.679918 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.680041 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:15:46.680437 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.681230 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.682686 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.683144 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.683287 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.683783 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:15:46.684266 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:15:46.684425 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.684481 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.684556 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:15:46.692987 13417 raft_consensus.cc:1261] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 1 FOLLOWER]: Refusing update from remote peer 1a230dfcfa8f41809495b7b1e605e873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:46.692987 13694 raft_consensus.cc:1261] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 1 FOLLOWER]: Refusing update from remote peer 1a230dfcfa8f41809495b7b1e605e873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:46.693974 13767 consensus_queue.cc:1035] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bb9d9b68e44a84b1de3134b282bfa1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:46.694283 13774 consensus_queue.cc:1035] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d791fd9476644099abd8fc0dde876b67" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:46.707018 13784 mvcc.cc:205] Tried to move back new op lower bound from 7333831256844013568 to 7333831256442593280. Current Snapshot: MvccSnapshot[applied={T|T < 7333831256844013568}]
I20260927 05:15:46.707846 13785 mvcc.cc:205] Tried to move back new op lower bound from 7333831256844013568 to 7333831256442593280. Current Snapshot: MvccSnapshot[applied={T|T < 7333831256844013568}]
I20260927 05:15:46.724128 13786 mvcc.cc:205] Tried to move back new op lower bound from 7333831256844013568 to 7333831256442593280. Current Snapshot: MvccSnapshot[applied={T|T < 7333831256844013568}]
I20260927 05:15:46.729018 13264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:15:46.729379 13264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:15:46.734416 13264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b0c7f26e745e39cba2fb59e1178ea: Sending DeleteTablet for 3 replicas of tablet 133b9e2d9816475999f6f588a49857bc
I20260927 05:15:46.735596 13397 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b9e2d9816475999f6f588a49857bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52828
I20260927 05:15:46.736840 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b9e2d9816475999f6f588a49857bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42704
I20260927 05:15:46.737602 13675 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b9e2d9816475999f6f588a49857bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42572
I20260927 05:15:46.739356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13302
I20260927 05:15:46.747766 13793 tablet_replica.cc:330] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: stopping tablet replica
I20260927 05:15:46.748219 13793 raft_consensus.cc:2229] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:15:46.748708 13795 tablet_replica.cc:330] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: stopping tablet replica
I20260927 05:15:46.748857 13793 raft_consensus.cc:2258] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:46.749205 13795 raft_consensus.cc:2229] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:15:46.749802 13795 raft_consensus.cc:2258] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:46.750934 13793 ts_tablet_manager.cc:1905] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:46.752143 13795 ts_tablet_manager.cc:1905] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:46.758240 13793 ts_tablet_manager.cc:1918] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:46.758417 13793 log.cc:1196] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/133b9e2d9816475999f6f588a49857bc
I20260927 05:15:46.759169 13793 ts_tablet_manager.cc:1939] T 133b9e2d9816475999f6f588a49857bc P d2bb9d9b68e44a84b1de3134b282bfa1: Deleting consensus metadata
I20260927 05:15:46.761008 13251 catalog_manager.cc:4841] TS d2bb9d9b68e44a84b1de3134b282bfa1 (127.0.97.1:38745): tablet 133b9e2d9816475999f6f588a49857bc (table test-table [id=79903cf183934d9a99dca217f57a4fae]) successfully deleted
I20260927 05:15:46.762038 13794 tablet_replica.cc:330] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: stopping tablet replica
I20260927 05:15:46.762429 13794 raft_consensus.cc:2229] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:15:46.763059 13794 raft_consensus.cc:2258] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:46.776734 13794 ts_tablet_manager.cc:1905] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:46.780664 13795 ts_tablet_manager.cc:1918] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:46.780964 13795 log.cc:1196] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/133b9e2d9816475999f6f588a49857bc
I20260927 05:15:46.781714 13795 ts_tablet_manager.cc:1939] T 133b9e2d9816475999f6f588a49857bc P d791fd9476644099abd8fc0dde876b67: Deleting consensus metadata
I20260927 05:15:46.783664 13252 catalog_manager.cc:4841] TS d791fd9476644099abd8fc0dde876b67 (127.0.97.3:33391): tablet 133b9e2d9816475999f6f588a49857bc (table test-table [id=79903cf183934d9a99dca217f57a4fae]) successfully deleted
I20260927 05:15:46.784790 13794 ts_tablet_manager.cc:1918] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:46.784996 13794 log.cc:1196] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/133b9e2d9816475999f6f588a49857bc
I20260927 05:15:46.785540 13794 ts_tablet_manager.cc:1939] T 133b9e2d9816475999f6f588a49857bc P 1a230dfcfa8f41809495b7b1e605e873: Deleting consensus metadata
I20260927 05:15:46.787343 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.740863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.0.97.1:38745)
0927 05:15:46.741085 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.741094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.759352 (+ 18258us) server_negotiation.cc:407] Connection header received
0927 05:15:46.759617 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.759621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.759702 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.759815 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:46.761686 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.762806 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.764910 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.765573 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.765661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.769020 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:46.769033 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:46.769050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:46.769085 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:46.773068 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.773898 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.773909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.773913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.774070 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.774475 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.774480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.774483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.774882 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:46.780841 (+ 5959us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.786679 (+ 5838us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.786938 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:15:46.788904 13457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:46.790427 13252 catalog_manager.cc:4841] TS 1a230dfcfa8f41809495b7b1e605e873 (127.0.97.2:41571): tablet 133b9e2d9816475999f6f588a49857bc (table test-table [id=79903cf183934d9a99dca217f57a4fae]) successfully deleted
I20260927 05:15:46.884943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13302
I20260927 05:15:46.902928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13466
I20260927 05:15:46.917797 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:46.904068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.0.97.2:41571)
0927 05:15:46.904249 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:46.904260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:46.904275 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:15:46.904326 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:46.904329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:46.904395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:46.904510 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:46.905908 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.906864 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.908163 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:46.908686 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:46.908808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:46.911650 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:46.911670 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:46.911683 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:46.911721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:46.914772 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.915785 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.915792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.915795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.915922 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:46.916281 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:46.916287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:46.916290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:46.916624 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:46.916960 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:46.917193 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 05:15:46.917402 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:15:46.918483 13601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:47.028091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13466
I20260927 05:15:47.045360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13610
I20260927 05:15:47.058386 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:47.046131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.97.3:33391)
0927 05:15:47.046326 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:47.046332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:47.046351 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:15:47.046542 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:47.046546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:47.046610 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:47.046709 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:47.048047 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:47.048728 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:47.050010 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:47.050429 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:47.050557 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:47.052998 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:47.053017 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:47.053033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:47.053076 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:47.055742 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:47.056361 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:47.056377 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:47.056380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:47.056502 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:47.056864 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:47.056870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:47.056873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:47.057220 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:47.057540 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:47.057843 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:15:47.058039 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:15:47.059072 13735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:47.144420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13610
I20260927 05:15:47.160818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13232
I20260927 05:15:47.161964 13295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:47.270749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13232
I20260927 05:15:47.292094 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:15:47.627242 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:15:47.627458 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:15:47.972326 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:15:47.972 UTC [12285] LOG: received smart shutdown request
2026-09-27 05:15:47.975 UTC [12285] LOG: background worker "logical replication launcher" (PID 12294) exited with exit code 1
2026-09-27 05:15:47.976 UTC [12289] LOG: shutting down
2026-09-27 05:15:48.128 UTC [12285] LOG: database system is shut down
2026-09-27T05:15:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50382 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:15:48 dist-test-slave-1t6p krb5kdc[13809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:15:48 dist-test-slave-1t6p krb5kdc[13809](info): set up 2 sockets
Sep 27 05:15:48 dist-test-slave-1t6p krb5kdc[13809](info): commencing operation
krb5kdc: starting...
W20260927 05:15:49.351224 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:15:49 dist-test-slave-1t6p krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:15:49.559152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44555
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:49.883311 13822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:49.883581 13822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:49.883677 13822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:49.890766 13822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:49.890864 13822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:49.890918 13822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:49.890973 13822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:49.891060 13822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:49.900862 13822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:49.902053 13822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:49.903286 13822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:49.914713 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:49.914856 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:49.914716 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:49.915999 13822 server_base.cc:1034] running on GCE node
I20260927 05:15:49.916705 13822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:49.917951 13822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:49.919234 13822 hybrid_clock.cc:648] HybridClock initialized: now 1790486149919189 us; error 43 us; skew 500 ppm
I20260927 05:15:49.919577 13822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:49 dist-test-slave-1t6p krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:49.922456 13822 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:49.923869 13822 webserver.cc:460] Webserver started at http://127.0.97.62:41353/ using document root <none> and password file <none>
I20260927 05:15:49.924504 13822 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:49.924642 13822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:49.924937 13822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:49.927837 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "3cab5018032143d8a6d713403ec21e30"
format_stamp: "Formatted at 2026-09-27 05:15:49 on dist-test-slave-1t6p"
server_key: "e470f401bc6ba493e8bade0f246755eb"
server_key_iv: "9085792dfbebce741b78bd1af16f57ab"
server_key_version: "encryptionkey@0"
I20260927 05:15:49.928591 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "3cab5018032143d8a6d713403ec21e30"
format_stamp: "Formatted at 2026-09-27 05:15:49 on dist-test-slave-1t6p"
server_key: "e470f401bc6ba493e8bade0f246755eb"
server_key_iv: "9085792dfbebce741b78bd1af16f57ab"
server_key_version: "encryptionkey@0"
I20260927 05:15:49.933414 13822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:15:49.936669 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:49.937896 13822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:15:49.938071 13822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "3cab5018032143d8a6d713403ec21e30"
format_stamp: "Formatted at 2026-09-27 05:15:49 on dist-test-slave-1t6p"
server_key: "e470f401bc6ba493e8bade0f246755eb"
server_key_iv: "9085792dfbebce741b78bd1af16f57ab"
server_key_version: "encryptionkey@0"
I20260927 05:15:49.938264 13822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:49.959929 13822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:49.963778 13822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:49.964054 13822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:49.982398 13822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44555
I20260927 05:15:49.982436 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44555 every 8 connection(s)
I20260927 05:15:49.983759 13822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:49.987318 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:49.988380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13822
I20260927 05:15:49.988603 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:15:49.989434 388 external_mini_cluster.cc:1428] Setting key ce5ade2b96418eb9c290f4250e4d7fc1
I20260927 05:15:49.997346 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:50.021106 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.000556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.0.97.62:44555)
0927 05:15:50.002226 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.002238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.002285 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:15:50.003794 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.003800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.004004 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.004207 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.005434 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.006606 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.008032 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.008580 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.008800 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.013856 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.013865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.013876 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.013907 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.016781 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.017479 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.017491 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.017494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.017646 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.018043 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.018054 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.018056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.018534 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.018740 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.019329 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.019635 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":1466,"thread_start_us":1254,"threads_started":1}
I20260927 05:15:50.021087 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.000547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44555 (local address 127.0.0.1:39810)
0927 05:15:50.001451 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.001502 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.003437 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.004219 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.004225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.004919 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.005242 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.005259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.006768 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.006773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.007894 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.007905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.008850 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.008857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.009095 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.009741 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.009766 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.013629 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.016935 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.016947 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.016959 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.017342 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.017770 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.017775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.017779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.017932 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.018686 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.018693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.018901 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.019191 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.019751 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":135,"thread_start_us":409,"threads_started":1}
I20260927 05:15:50.024294 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:15:50.025372 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.024866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36216)
0927 05:15:50.025065 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.025197 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 05:15:50.025630 13839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
I20260927 05:15:50.027241 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.026774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36224)
0927 05:15:50.026947 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.027072 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:15:50.027426 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42235: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:15:50.038771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42235
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:15:50.076793 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } attempt: 1
I20260927 05:15:50.077987 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.077526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36238)
0927 05:15:50.077732 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.077831 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:15:50.079164 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.078724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36248)
0927 05:15:50.078884 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.078963 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:15:50.079349 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42235: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
I20260927 05:15:50.147578 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } attempt: 2
I20260927 05:15:50.148787 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.148217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36250)
0927 05:15:50.148483 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.148608 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 05:15:50.149825 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.149428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36262)
0927 05:15:50.149569 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.149663 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 05:15:50.150014 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42235: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
I20260927 05:15:50.241285 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } attempt: 3
I20260927 05:15:50.242524 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.241942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36272)
0927 05:15:50.242191 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.242322 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:15:50.243659 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.243267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36286)
0927 05:15:50.243422 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.243513 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:15:50.243847 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42235: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
W20260927 05:15:50.352815 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:50.353070 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:50.353248 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:50.360112 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:50.360230 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:50.360311 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:50.360411 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:50.360494 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:50.369700 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:50.370898 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:50.372141 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:50.383145 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:50.383183 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:50.383323 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:50.384718 13896 server_base.cc:1034] running on GCE node
I20260927 05:15:50.385406 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:50.386667 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:50.387087 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } attempt: 4
I20260927 05:15:50.387880 13896 hybrid_clock.cc:648] HybridClock initialized: now 1790486150387826 us; error 46 us; skew 500 ppm
I20260927 05:15:50.388163 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.387626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36290)
0927 05:15:50.387897 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.388011 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 05:15:50.388264 13896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:15:50.389128 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.388751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36296)
0927 05:15:50.388890 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.388962 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:15:50.389334 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42235: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:50.396425 13896 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:15:50.397801 13896 webserver.cc:460] Webserver started at http://127.0.97.61:40175/ using document root <none> and password file <none>
I20260927 05:15:50.398456 13896 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:50.398594 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:50.398886 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:50.401688 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data/instance:
uuid: "064b40445e2042b3b7014aeff8bda917"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "c1b150cc83c74aa883a3c38ff6c60882"
server_key_iv: "029601cf46a0dcd7ccac33a9a997da0b"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.402434 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal/instance:
uuid: "064b40445e2042b3b7014aeff8bda917"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "c1b150cc83c74aa883a3c38ff6c60882"
server_key_iv: "029601cf46a0dcd7ccac33a9a997da0b"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.407315 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 05:15:50.410580 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:50.411859 13896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:50.412034 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "064b40445e2042b3b7014aeff8bda917"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "c1b150cc83c74aa883a3c38ff6c60882"
server_key_iv: "029601cf46a0dcd7ccac33a9a997da0b"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.412231 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:50.456920 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:50.461148 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:50.461432 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:50.479262 13896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42235
I20260927 05:15:50.479295 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42235 every 8 connection(s)
I20260927 05:15:50.480556 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:15:50.484156 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:50.488538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13896
I20260927 05:15:50.488718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-1/wal/instance
I20260927 05:15:50.489272 388 external_mini_cluster.cc:1428] Setting key eb9b7ae6a9ed6082a989e9a5dcec22a8
I20260927 05:15:50.491575 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 05:15:50.513253 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.494810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.97.62:44555)
0927 05:15:50.495031 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.495039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.496401 (+ 1362us) server_negotiation.cc:407] Connection header received
0927 05:15:50.498490 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.498496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.498590 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.498703 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.500244 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.500916 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.502437 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.502854 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.502990 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.506422 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.506433 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.506442 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.506492 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.510167 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.510912 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.510917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.510920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.511112 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.511563 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.511568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.511572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.511847 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.512012 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.512715 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.512928 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:15:50.514123 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.495376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44555 (local address 127.0.0.1:39816)
0927 05:15:50.496234 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.496289 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.498213 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.498942 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.498949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.499506 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.500046 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.500061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.501082 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.501086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.502283 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.502295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.503114 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.503121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.503345 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.504023 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.504045 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.506163 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.510349 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.510355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.510367 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.510778 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.511249 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.511254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.511258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.511443 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.511987 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.511995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.512356 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.512605 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.513091 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":379,"threads_started":1}
I20260927 05:15:50.515792 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:15:50.531277 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.516681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36302)
0927 05:15:50.516865 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.516883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.517018 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.517709 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.517713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.518119 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.518235 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.518242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.519960 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.519964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.520789 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.520800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.522014 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.522019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.522171 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.522736 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.522751 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.524665 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.528929 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.528932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.528935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.529265 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.529674 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.529677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.529679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.529816 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.530572 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.530575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.530674 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.530784 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.530947 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:15:50.531708 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.516768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.0.97.61:42235)
0927 05:15:50.517339 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.517344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.517365 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:15:50.517402 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.517407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.517560 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.517706 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.518375 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.519806 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.520922 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.521859 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.522004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.524886 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.524900 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.524919 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.524959 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.528807 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.529393 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.529402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.529406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.529566 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.529923 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.529931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.529936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.530439 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.530656 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.530954 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.531264 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":271,"threads_started":1}
I20260927 05:15:50.533864 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:15:50.534909 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.534414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34110)
0927 05:15:50.534583 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.534759 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:15:50.535169 13914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.536595 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.536264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34122)
0927 05:15:50.536383 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.536502 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:15:50.536820 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:15:50.542066 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39503
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:15:50.586112 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 1
I20260927 05:15:50.587347 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.586876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34124)
0927 05:15:50.587104 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.587204 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:15:50.588418 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.588113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34130)
0927 05:15:50.588244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.588320 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:15:50.588821 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.657034 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 2
I20260927 05:15:50.658207 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.657781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34138)
0927 05:15:50.657967 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.658059 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:15:50.659322 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.658971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34152)
0927 05:15:50.659145 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.659213 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 05:15:50.659547 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.688576 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } attempt: 5
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:15:50.701748 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.689438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.0.97.61:42235)
0927 05:15:50.689609 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.689617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.689661 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:15:50.689901 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.689907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.689985 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.690093 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.690903 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.691670 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.692859 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.693291 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.693427 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.696618 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.696623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.696627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.696657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.699262 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.699893 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.699901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.699904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.700026 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.700393 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.700399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.700403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.700669 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.700831 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.701204 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.701418 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:15:50.701753 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.689353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36308)
0927 05:15:50.689572 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.689594 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.689749 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.690131 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.690137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.690551 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.690742 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.690752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.691837 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.691842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.692674 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.692718 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.693440 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.693446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.693639 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.694395 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.694428 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.696426 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.699386 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.699393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.699397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.699769 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.700140 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.700145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.700148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.700286 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.700795 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.700800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.700951 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.701099 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.701381 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:15:50.702988 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:15:50.704007 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.703656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34160)
0927 05:15:50.703808 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.703879 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:15:50.704906 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.704590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34170)
0927 05:15:50.704735 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.704815 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:15:50.705129 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.750769 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 3
I20260927 05:15:50.751963 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.751493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34186)
0927 05:15:50.751720 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.751807 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 05:15:50.753000 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.752584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34194)
0927 05:15:50.752816 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.752889 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 05:15:50.753185 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.756323 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 1
I20260927 05:15:50.757287 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.756900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34210)
0927 05:15:50.757061 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.757144 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:15:50.758239 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.757931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34222)
0927 05:15:50.758081 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.758141 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:15:50.758440 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.826654 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 2
I20260927 05:15:50.827858 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.827399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34232)
0927 05:15:50.827617 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.827702 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 05:15:50.828887 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.828547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34234)
0927 05:15:50.828697 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.828783 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:15:50.829089 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
W20260927 05:15:50.861174 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:50.861506 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:50.861680 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:50.868785 13970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:50.868875 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:50.868919 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:50.868968 13970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:50.869004 13970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:50.878312 13970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44555,127.0.97.61:42235,127.0.97.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:50.879520 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:50.880734 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:50.892210 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:50.892313 13978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:50.892354 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:50.893119 13970 server_base.cc:1034] running on GCE node
I20260927 05:15:50.893827 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:50.895248 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:50.896381 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 4
I20260927 05:15:50.896463 13970 hybrid_clock.cc:648] HybridClock initialized: now 1790486150896416 us; error 63 us; skew 500 ppm
I20260927 05:15:50.896903 13970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:15:50.897624 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.896976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34250)
0927 05:15:50.897335 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.897430 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 05:15:50.898628 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.898242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34252)
0927 05:15:50.898409 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.898495 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:15:50.898835 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:50.900463 13970 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:15:50.901875 13970 webserver.cc:460] Webserver started at http://127.0.97.60:44329/ using document root <none> and password file <none>
I20260927 05:15:50.902532 13970 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:50.902628 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:50.902952 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:50.905790 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data/instance:
uuid: "6d1162cdd4df4eca91d3fc20b05e93ce"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "a325bd37a59526735c596dd2b3879629"
server_key_iv: "089e31c536d390ba724c4401df954453"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.906548 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal/instance:
uuid: "6d1162cdd4df4eca91d3fc20b05e93ce"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "a325bd37a59526735c596dd2b3879629"
server_key_iv: "089e31c536d390ba724c4401df954453"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.911432 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 05:15:50.914516 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:50.915845 13970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:50.916011 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "6d1162cdd4df4eca91d3fc20b05e93ce"
format_stamp: "Formatted at 2026-09-27 05:15:50 on dist-test-slave-1t6p"
server_key: "a325bd37a59526735c596dd2b3879629"
server_key_iv: "089e31c536d390ba724c4401df954453"
server_key_version: "encryptionkey@0"
I20260927 05:15:50.916208 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:50.926455 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:50.929978 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:50.930265 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:50.935271 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 3
I20260927 05:15:50.936456 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.935921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34264)
0927 05:15:50.936192 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.936303 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 05:15:50.937618 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.937252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34270)
0927 05:15:50.937454 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.937519 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 05:15:50.937922 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39503: connect: Connection refused (error 111)
I20260927 05:15:50.948523 13970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39503
I20260927 05:15:50.948558 14036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39503 every 8 connection(s)
I20260927 05:15:50.949795 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:15:50.950825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 13970
I20260927 05:15:50.951033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485951045978-388-0/minicluster-data/master-2/wal/instance
I20260927 05:15:50.951565 388 external_mini_cluster.cc:1428] Setting key 890f971d8fbf0c59767347f899adbc03
I20260927 05:15:50.954299 14037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:50.962541 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:50.975634 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.953331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.0.97.60:39503)
0927 05:15:50.954066 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.954080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.954128 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:15:50.956337 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.956362 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.956802 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.957218 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.958493 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.960567 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.961976 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.962973 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.963201 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.967746 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.967754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.967763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.967800 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.971329 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.972199 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.972206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.972211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.972381 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.972801 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.972805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.972807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.973365 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.973649 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.974027 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.974467 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260927 05:15:50.980280 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.965100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44555 (local address 127.0.0.1:39824)
0927 05:15:50.965880 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.965918 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.966071 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.966541 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.966550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.967368 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.967743 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.967756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.969119 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.969123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.970214 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.970225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.971230 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.971235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.971349 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.971922 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.971952 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.974555 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.977474 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.977483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.977488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.977988 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.978376 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.978384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.978391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.978540 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.979276 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.979287 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.979476 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.979696 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.980019 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":426,"threads_started":1}
I20260927 05:15:50.980476 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.965136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.97.62:44555)
0927 05:15:50.965357 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.965365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.966023 (+ 658us) server_negotiation.cc:407] Connection header received
0927 05:15:50.966211 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.966218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.966307 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.966408 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.967905 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.968958 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.970354 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.971053 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.971188 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.974749 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.974753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.974757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.974789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.977352 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.978088 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.978091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.978094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.978248 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.978686 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.978699 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.978703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.979119 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.979306 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.979838 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.980079 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:15:50.981671 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:15:50.995733 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.978809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.97.62:44555)
0927 05:15:50.979487 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.979495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.979519 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:15:50.979565 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.979570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.979639 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.979741 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.981378 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.982608 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.984425 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.985053 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.985193 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.989427 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.989435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.989439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.989476 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.993129 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.993742 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.993748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.993751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.993897 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.994278 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.994281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.994283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.994526 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.994741 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.995166 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.995378 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":322,"threads_started":1}
I20260927 05:15:50.996654 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.982412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:36316)
0927 05:15:50.982620 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.982638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.982734 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.983347 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.983351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.983690 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.983801 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.983809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.984967 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.984973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.986342 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.986359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.987453 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.987459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:50.987666 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.988241 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:50.988269 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:50.990698 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:50.994199 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.994207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.994213 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.994604 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.995055 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:50.995060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:50.995064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:50.995251 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:50.995795 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:50.995799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:50.995939 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 05:15:50.996072 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:15:50.996322 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 05:15:50.996735 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.982870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.0.97.61:42235)
0927 05:15:50.983102 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.983107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.983120 (+ 13us) server_negotiation.cc:407] Connection header received
0927 05:15:50.983165 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.983168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.983223 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.983330 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.983963 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.984803 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.986510 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:50.987281 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:50.987416 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:50.990906 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:50.990913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:50.990921 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:50.990959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:50.994084 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.994732 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.994737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.994741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.994879 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:50.995356 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:50.995363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:50.995367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:50.995693 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:50.995814 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:50.996172 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 05:15:50.996385 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:15:50.997567 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:15:50 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:15:51.010598 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.998302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.0.97.60:39503)
0927 05:15:50.998459 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.998467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.998510 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:15:50.998559 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.998565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.998636 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.998760 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.999430 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.000145 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.001310 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.001812 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.001901 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.004840 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.004845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.004848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.004875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.007916 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.008603 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.008610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.008613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.008759 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.009124 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.009129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.009132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.009482 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.009631 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.009933 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.010267 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":53}
I20260927 05:15:51.010604 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.998206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34292)
0927 05:15:50.998342 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:50.998359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:50.998442 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:50.998764 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:50.998767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:50.999170 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:50.999290 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:50.999298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.000270 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.000276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.001164 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:51.001183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.001968 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.001973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.002149 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.002728 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:51.002755 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:51.004654 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:51.008062 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.008070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.008076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.008492 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.008899 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.008902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.008905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.009017 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.009595 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:51.009598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:51.009702 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 05:15:51.009803 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:15:51.010107 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 05:15:51.012729 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:50.997705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.97.61:42235)
0927 05:15:50.997851 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:50.997859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:50.997874 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:15:50.997917 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:50.997921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:50.997990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:50.998076 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:50.999604 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.000310 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.001532 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.001983 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.002100 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.006343 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.006364 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.006371 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.006416 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.009770 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.010575 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.010582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.010586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.010747 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.011256 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.011261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.011263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.011587 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.011911 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.012188 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.012411 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:15:51.016439 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: Bootstrap starting.
I20260927 05:15:51.017392 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:15:51.019927 14037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:51.021298 14037 log.cc:824] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:51.024906 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: No bootstrap required, opened a new log
I20260927 05:15:51.038663 14037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.039067 14037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:51.039176 14037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1162cdd4df4eca91d3fc20b05e93ce, State: Initialized, Role: FOLLOWER
I20260927 05:15:51.039937 14037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.043875 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.023018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.0.97.62:44555)
0927 05:15:51.023200 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:51.023206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:51.023316 (+ 110us) server_negotiation.cc:407] Connection header received
0927 05:15:51.023617 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:51.023627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:51.023746 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:51.023837 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:51.027192 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.028539 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.031248 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.032149 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.032266 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.036597 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.036603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.036607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.036641 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.040254 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.041105 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.041111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.041115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.041250 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.041778 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.041785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.041790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.042121 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.042356 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.042896 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.043125 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 05:15:51.046974 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.047420 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.048871 14037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:51.053048 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.023240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.97.61:42235)
0927 05:15:51.028380 (+ 5140us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:51.028386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:51.035344 (+ 6958us) server_negotiation.cc:407] Connection header received
0927 05:15:51.036455 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:51.036462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:51.036543 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:51.036608 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:51.038532 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.039678 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.041846 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.042518 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.042609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.045991 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.045998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.046003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.046041 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.049475 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.050307 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.050313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.050316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.050517 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.051074 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.051082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.051088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.051491 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.051712 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.052335 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.052605 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5016}
I20260927 05:15:51.054545 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.024225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.0.97.60:39503)
0927 05:15:51.031182 (+ 6957us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:51.031201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:51.035902 (+ 4701us) server_negotiation.cc:407] Connection header received
0927 05:15:51.036061 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:51.036069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:51.036154 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:51.036254 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:51.038123 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.039155 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.042028 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.043099 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.043239 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.047661 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.047666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.047671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.047731 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.051172 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.052044 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.052048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.052050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.052184 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.052646 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.052653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.052658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.053071 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.053320 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.054170 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.054326 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6788}
W20260927 05:15:51.061566 14062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:15:51.061697 14062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:15:51.057875 13970 master_runner.cc:186] Error getting master registration for 127.0.97.62:44555: OK, instance_id { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" instance_seqno: 1790486149976996 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:15:51.067116 13970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:15:51.115156 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 4
Sep 27 05:15:51 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:15:51.129716 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.115705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34308)
0927 05:15:51.115962 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:51.115977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:51.116066 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:51.116391 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:51.116395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:51.116896 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:51.117059 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:51.117068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.118557 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.118563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.120088 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:51.120103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.120944 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.120950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.121144 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.121740 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:51.121764 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:51.124172 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:51.127325 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.127331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.127335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.127692 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.128072 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.128078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.128082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.128231 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.128771 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:51.128775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:51.128908 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 05:15:51.129041 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:15:51.129310 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 05:15:51.129769 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.115853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.0.97.60:39503)
0927 05:15:51.116037 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:51.116043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:51.116063 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:15:51.116169 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:51.116175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:51.116262 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:51.116362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:51.117192 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.118391 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.120249 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.120797 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.120903 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.124345 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.124357 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.124361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.124408 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.127182 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.127811 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.127816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.127820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.127955 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.128348 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.128353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.128357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.128640 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.128790 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.129147 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.129385 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:15:51.135354 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: Bootstrap starting.
I20260927 05:15:51.139329 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:51.140573 13889 log.cc:824] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:51.143821 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: No bootstrap required, opened a new log
I20260927 05:15:51.154392 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.154740 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:51.154848 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cab5018032143d8a6d713403ec21e30, State: Initialized, Role: FOLLOWER
I20260927 05:15:51.155449 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.156252 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.156551 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.157512 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:15:51.167513 14076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:15:51.167701 14076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:15:51.166862 13822 master_runner.cc:186] Error getting master registration for 127.0.97.61:42235: OK, instance_id { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" instance_seqno: 1790486150474149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:15:51.177932 13822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 05:15:51.182193 13991 tablet.cc:2369] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:51.198056 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } attempt: 5
Sep 27 05:15:51 dist-test-slave-1t6p krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:15:51.214766 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.198574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39503 (local address 127.0.0.1:34320)
0927 05:15:51.198820 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:51.198840 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:51.198940 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:51.199334 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:51.199340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:51.199869 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:51.200075 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:51.200086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.201524 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.201530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.202966 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:51.202982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.204067 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.204073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:51.204280 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.205001 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:51.205024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:51.207885 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:51.211988 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.211994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.211998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.212506 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.212932 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:51.212938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:51.212941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:51.213106 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:51.213732 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:51.213737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:51.213887 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 05:15:51.214040 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:15:51.214319 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 05:15:51.214803 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:51.198707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.97.60:39503)
0927 05:15:51.198877 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:51.198883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:51.198902 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:15:51.199103 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:51.199111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:51.199186 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:51.199289 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:51.200235 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.201376 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.203186 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:51.203916 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:51.204042 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:51.208079 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:51.208085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:51.208090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:51.208133 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:51.211842 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.212653 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.212660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.212663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.212808 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:51.213234 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:51.213239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:51.213243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:51.213606 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:51.213766 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:51.214187 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 05:15:51.214406 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 05:15:51.216781 13843 tablet.cc:2369] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:51.221392 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: Bootstrap starting.
I20260927 05:15:51.225339 13963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:51.226583 13963 log.cc:824] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:51.229668 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: No bootstrap required, opened a new log
I20260927 05:15:51.234447 14051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:15:51.234726 14051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.236238 14051 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cab5018032143d8a6d713403ec21e30 (127.0.97.62:44555), 064b40445e2042b3b7014aeff8bda917 (127.0.97.61:42235)
I20260927 05:15:51.237571 13864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cab5018032143d8a6d713403ec21e30" is_pre_election: true
I20260927 05:15:51.238181 13864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1162cdd4df4eca91d3fc20b05e93ce in term 0.
I20260927 05:15:51.239145 13988 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cab5018032143d8a6d713403ec21e30, 6d1162cdd4df4eca91d3fc20b05e93ce; no voters:
I20260927 05:15:51.239562 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:15:51.239698 14051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:15:51.239779 14051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:51.240993 13963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.241356 13963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:51.241475 13963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064b40445e2042b3b7014aeff8bda917, State: Initialized, Role: FOLLOWER
I20260927 05:15:51.242142 13963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.244952 14051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.246904 13864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cab5018032143d8a6d713403ec21e30"
I20260927 05:15:51.247175 13864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:51.251544 13864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1162cdd4df4eca91d3fc20b05e93ce in term 1.
I20260927 05:15:51.252065 13988 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cab5018032143d8a6d713403ec21e30, 6d1162cdd4df4eca91d3fc20b05e93ce; no voters:
I20260927 05:15:51.252830 14080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:51.253484 14051 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [CANDIDATE]: Term 1 election: Requested vote from peers 3cab5018032143d8a6d713403ec21e30 (127.0.97.62:44555), 064b40445e2042b3b7014aeff8bda917 (127.0.97.61:42235)
I20260927 05:15:51.255223 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.255978 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.258188 13963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:51.258325 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064b40445e2042b3b7014aeff8bda917"
I20260927 05:15:51.258800 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:51.262629 13937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1162cdd4df4eca91d3fc20b05e93ce in term 1.
I20260927 05:15:51.263492 14080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [term 1 LEADER]: Becoming Leader. State: Replica: 6d1162cdd4df4eca91d3fc20b05e93ce, State: Running, Role: LEADER
I20260927 05:15:51.263599 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064b40445e2042b3b7014aeff8bda917" is_pre_election: true
I20260927 05:15:51.263903 13938 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d1162cdd4df4eca91d3fc20b05e93ce in term 1. Re-sending same reply.
I20260927 05:15:51.264282 14080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } }
I20260927 05:15:51.267735 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1162cdd4df4eca91d3fc20b05e93ce. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.268091 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:51.278918 14085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:51.281558 14085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:51.295939 13937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [term 1 FOLLOWER]: Refusing update from remote peer 6d1162cdd4df4eca91d3fc20b05e93ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:51.297009 14051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:51.304925 13864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [term 1 FOLLOWER]: Refusing update from remote peer 6d1162cdd4df4eca91d3fc20b05e93ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:51.306056 14051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:51.335315 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1162cdd4df4eca91d3fc20b05e93ce. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.335628 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.338972 14088 mvcc.cc:205] Tried to move back new op lower bound from 7333831275697246208 to 7333831275582861312. Current Snapshot: MvccSnapshot[applied={T|T < 7333831275697246208}]
I20260927 05:15:51.344439 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1162cdd4df4eca91d3fc20b05e93ce. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.344795 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.345451 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.346037 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:51.346310 14094 mvcc.cc:205] Tried to move back new op lower bound from 7333831275697246208 to 7333831275582861312. Current Snapshot: MvccSnapshot[applied={T|T < 7333831275697246208}]
I20260927 05:15:51.347817 14085 catalog_manager.cc:1347] Generated new cluster ID: c61e86b0bd0a47869157c52d442284cf
I20260927 05:15:51.347924 14085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:51.355832 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.356171 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:51.368888 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.369095 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 05:15:51.373071 14101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:15:51.373208 14101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:15:51.388468 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cab5018032143d8a6d713403ec21e30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44555 } } peers { permanent_uuid: "064b40445e2042b3b7014aeff8bda917" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42235 } } peers { permanent_uuid: "6d1162cdd4df4eca91d3fc20b05e93ce" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39503 } } }
I20260927 05:15:51.388727 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:15:51.400738 14085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:15:51.404366 14085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:51.427989 14085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce: Generated new TSK 0
I20260927 05:15:51.429090 14085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:15:52.169713 14076 catalog_manager.cc:1259] Loaded cluster ID: c61e86b0bd0a47869157c52d442284cf
I20260927 05:15:52.169813 14076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: loading cluster ID for follower catalog manager: success
I20260927 05:15:52.172180 14076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: acquiring CA information for follower catalog manager: success
I20260927 05:15:52.173835 14076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cab5018032143d8a6d713403ec21e30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:15:52.375157 14101 catalog_manager.cc:1259] Loaded cluster ID: c61e86b0bd0a47869157c52d442284cf
I20260927 05:15:52.375257 14101 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: loading cluster ID for follower catalog manager: success
I20260927 05:15:52.377622 14101 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: acquiring CA information for follower catalog manager: success
I20260927 05:15:52.379307 14101 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 064b40445e2042b3b7014aeff8bda917: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:15:52.577268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13822
I20260927 05:15:52.579121 13884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:52.687708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13822
I20260927 05:15:52.708137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13896
I20260927 05:15:52.709228 13958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:52.772081 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:52.771326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44555 (local address 127.0.0.1:35632)
0927 05:15:52.771793 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:52.771904 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":262,"threads_started":1}
W20260927 05:15:52.772410 13988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44555: connect: Connection refused (error 111)
I20260927 05:15:52.773749 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:52.773426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44555 (local address 127.0.0.1:35642)
0927 05:15:52.773560 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:52.773635 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:15:52.774027 13988 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce -> Peer 3cab5018032143d8a6d713403ec21e30 (127.0.97.62:44555): Couldn't send request to peer 3cab5018032143d8a6d713403ec21e30. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44555: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:15:52.814320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13896
W20260927 05:15:52.834003 13987 connection.cc:535] client connection to 127.0.97.61:42235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 05:15:52.834260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 13970
I20260927 05:15:52.835285 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:52.834797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42235 (local address 127.0.0.1:42698)
0927 05:15:52.834993 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:52.835099 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 05:15:52.835394 14032 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 05:15:52.835466 13987 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d1162cdd4df4eca91d3fc20b05e93ce -> Peer 064b40445e2042b3b7014aeff8bda917 (127.0.97.61:42235): Couldn't send request to peer 064b40445e2042b3b7014aeff8bda917. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:15:52.941154 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 13970
2026-09-27T05:15:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4688 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:15:53 dist-test-slave-1t6p krb5kdc[14115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:15:53 dist-test-slave-1t6p krb5kdc[14115](info): set up 2 sockets
Sep 27 05:15:53 dist-test-slave-1t6p krb5kdc[14115](info): commencing operation
krb5kdc: starting...
W20260927 05:15:54.035519 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:15:54.248047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43599
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:54.560839 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:54.561103 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:54.561271 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:54.568331 14128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:54.568423 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:54.568518 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:54.568622 14128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:54.568702 14128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:54.577992 14128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:54.579216 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:54.580473 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:54.592222 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:54.592299 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:54.592739 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:54.593176 14128 server_base.cc:1034] running on GCE node
I20260927 05:15:54.593907 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:54.595227 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:54.596446 14128 hybrid_clock.cc:648] HybridClock initialized: now 1790486154596400 us; error 54 us; skew 500 ppm
I20260927 05:15:54.596771 14128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:54.599779 14128 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:54.601156 14128 webserver.cc:460] Webserver started at http://127.0.97.62:36699/ using document root <none> and password file <none>
I20260927 05:15:54.601799 14128 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:54.601971 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:54.602269 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:54.605069 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "5d16b88ff7a24715af96671410363372"
format_stamp: "Formatted at 2026-09-27 05:15:54 on dist-test-slave-1t6p"
server_key: "d4cb497f2363be7e29ac3b4c3ee1f2e4"
server_key_iv: "05463a31475746bd297c21a76d6ad1e7"
server_key_version: "encryptionkey@0"
I20260927 05:15:54.605808 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d16b88ff7a24715af96671410363372"
format_stamp: "Formatted at 2026-09-27 05:15:54 on dist-test-slave-1t6p"
server_key: "d4cb497f2363be7e29ac3b4c3ee1f2e4"
server_key_iv: "05463a31475746bd297c21a76d6ad1e7"
server_key_version: "encryptionkey@0"
I20260927 05:15:54.610577 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:15:54.613804 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:54.615049 14128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:15:54.615222 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "5d16b88ff7a24715af96671410363372"
format_stamp: "Formatted at 2026-09-27 05:15:54 on dist-test-slave-1t6p"
server_key: "d4cb497f2363be7e29ac3b4c3ee1f2e4"
server_key_iv: "05463a31475746bd297c21a76d6ad1e7"
server_key_version: "encryptionkey@0"
I20260927 05:15:54.615410 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:54.641084 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:54.645404 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:54.645722 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:54.663918 14128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43599
I20260927 05:15:54.663961 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43599 every 8 connection(s)
I20260927 05:15:54.665225 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:54.665822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14128
I20260927 05:15:54.666044 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:15:54.666831 388 external_mini_cluster.cc:1428] Setting key fee16355094994540386116614cbd8ce
I20260927 05:15:54.669389 14195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:54.679647 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372: Bootstrap starting.
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:54.683214 14195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:54.684217 14195 log.cc:824] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:54.687161 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372: No bootstrap required, opened a new log
I20260927 05:15:54.688611 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:54.668609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.97.62:43599)
0927 05:15:54.669366 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:54.669380 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:54.669419 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:15:54.671206 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:54.671226 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:54.671571 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:54.671964 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:54.673395 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:54.674733 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:54.676102 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:54.676752 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:54.676971 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:54.681624 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:54.681638 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:54.681651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:54.681688 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:54.684855 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:54.685577 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:54.685583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:54.685591 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:54.685735 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:54.686120 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:54.686122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:54.686124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:54.686579 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:54.686794 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:54.687313 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 05:15:54.687707 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260927 05:15:54.692502 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:15:54.696810 14195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } }
I20260927 05:15:54.697106 14195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:54.697180 14195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d16b88ff7a24715af96671410363372, State: Initialized, Role: FOLLOWER
I20260927 05:15:54.697614 14195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } }
I20260927 05:15:54.697799 14195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:15:54.697898 14195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:15:54.698009 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:54.700004 14195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } }
I20260927 05:15:54.700418 14195 leader_election.cc:302] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d16b88ff7a24715af96671410363372; no voters:
I20260927 05:15:54.700973 14195 leader_election.cc:288] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:15:54.701133 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:54.701833 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [term 1 LEADER]: Becoming Leader. State: Replica: 5d16b88ff7a24715af96671410363372, State: Running, Role: LEADER
I20260927 05:15:54.702467 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } }
I20260927 05:15:54.703070 14195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:54.705175 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d16b88ff7a24715af96671410363372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } } }
I20260927 05:15:54.705305 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d16b88ff7a24715af96671410363372. Latest consensus state: current_term: 1 leader_uuid: "5d16b88ff7a24715af96671410363372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d16b88ff7a24715af96671410363372" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43599 } } }
I20260927 05:15:54.705585 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:54.705585 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372 [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:54.707309 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:54.712325 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:54.720276 14208 catalog_manager.cc:1347] Generated new cluster ID: 98f38ce74ce5497eaa0f5af2f8f19e4a
I20260927 05:15:54.720381 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:54.729533 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:15:54.730661 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:54.741580 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d16b88ff7a24715af96671410363372: Generated new TSK 0
I20260927 05:15:54.742295 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:15:54.753464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14128
I20260927 05:15:54.754503 14190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:54.843864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14128
2026-09-27T05:15:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1898 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14223](info): set up 2 sockets
Sep 27 05:15:54 dist-test-slave-1t6p krb5kdc[14223](info): commencing operation
krb5kdc: starting...
W20260927 05:15:55.935443 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:15:56 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:15:56.164705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:56.477239 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:56.477494 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:56.477689 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:56.484800 14236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:56.484894 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:56.484997 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:56.485111 14236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:56.485201 14236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:56.494746 14236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:56.496034 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:56.497267 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:56.508822 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:56.508808 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:56.508802 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:56.510008 14236 server_base.cc:1034] running on GCE node
I20260927 05:15:56.510720 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:56.512069 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:56.513307 14236 hybrid_clock.cc:648] HybridClock initialized: now 1790486156513229 us; error 63 us; skew 500 ppm
I20260927 05:15:56.513679 14236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:56 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:56.516871 14236 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:56.518319 14236 webserver.cc:460] Webserver started at http://127.0.97.62:44291/ using document root <none> and password file <none>
I20260927 05:15:56.518997 14236 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:56.519189 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:56.519487 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:56.522307 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "012b4dc648a14bb9a38600ecd433dcde"
format_stamp: "Formatted at 2026-09-27 05:15:56 on dist-test-slave-1t6p"
server_key: "388fa87410a643f883bba383210b31e2"
server_key_iv: "4ae817edaf3488df03bd701330ee13d4"
server_key_version: "encryptionkey@0"
I20260927 05:15:56.523218 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "012b4dc648a14bb9a38600ecd433dcde"
format_stamp: "Formatted at 2026-09-27 05:15:56 on dist-test-slave-1t6p"
server_key: "388fa87410a643f883bba383210b31e2"
server_key_iv: "4ae817edaf3488df03bd701330ee13d4"
server_key_version: "encryptionkey@0"
I20260927 05:15:56.528105 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:15:56.531358 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:56.532627 14236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:15:56.532817 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "012b4dc648a14bb9a38600ecd433dcde"
format_stamp: "Formatted at 2026-09-27 05:15:56 on dist-test-slave-1t6p"
server_key: "388fa87410a643f883bba383210b31e2"
server_key_iv: "4ae817edaf3488df03bd701330ee13d4"
server_key_version: "encryptionkey@0"
I20260927 05:15:56.532974 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:56.563211 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:56.567441 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:56.567675 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:56.586319 14236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40991
I20260927 05:15:56.586351 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40991 every 8 connection(s)
I20260927 05:15:56.587692 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:56.591810 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:56.593948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14236
I20260927 05:15:56.594180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:15:56.594964 388 external_mini_cluster.cc:1428] Setting key 12a5825e3a8c69d2a99189a90b211bc8
I20260927 05:15:56.605597 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap starting.
I20260927 05:15:56.609369 14303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:56.610657 14303 log.cc:824] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:15:56 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:56.613613 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: No bootstrap required, opened a new log
I20260927 05:15:56.619300 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:56.597101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.0.97.62:40991)
0927 05:15:56.598436 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:56.598451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:56.598501 (+ 50us) server_negotiation.cc:407] Connection header received
0927 05:15:56.600376 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:56.600399 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:56.600770 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:56.601069 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:56.602821 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:56.604289 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:56.606223 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:56.607185 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:56.607322 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:56.612347 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:56.612355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:56.612368 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:56.612401 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:56.615484 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:56.616159 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:56.616171 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:56.616180 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:56.616338 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:56.616729 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:56.616736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:56.616743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:56.617226 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:56.617525 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:56.617939 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 05:15:56.618319 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":617,"threads_started":1}
I20260927 05:15:56.621315 14303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:56.621663 14303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:56.621773 14303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Initialized, Role: FOLLOWER
I20260927 05:15:56.622344 14303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:56.622535 14303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:15:56.622638 14303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:15:56.622749 14303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:56.624998 14303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:56.625459 14303 leader_election.cc:302] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012b4dc648a14bb9a38600ecd433dcde; no voters:
I20260927 05:15:56.625988 14303 leader_election.cc:288] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:15:56.626173 14308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:56.626879 14308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 LEADER]: Becoming Leader. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Running, Role: LEADER
I20260927 05:15:56.627382 14308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:56.628213 14303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:56.630369 14310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012b4dc648a14bb9a38600ecd433dcde. Latest consensus state: current_term: 1 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:15:56.630330 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:15:56.630748 14310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:56.633319 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:56.633960 14316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:56.639670 14316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:56.647114 14316 catalog_manager.cc:1347] Generated new cluster ID: 0de358c8977146628782bb2d31ccb74c
I20260927 05:15:56.647223 14316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:56.656410 14316 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:15:56.657521 14316 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:56.667968 14316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Generated new TSK 0
I20260927 05:15:56.668694 14316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:15:56.725967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:57.045238 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:57.045488 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:57.045706 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:57.052522 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:57.052644 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:57.052830 14329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:15:57.062383 14329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:57.063561 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:57.064795 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:57.077200 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:57.077354 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:57.077320 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:57.078187 14329 server_base.cc:1034] running on GCE node
I20260927 05:15:57.078763 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:57.079696 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:57.080899 14329 hybrid_clock.cc:648] HybridClock initialized: now 1790486157080871 us; error 43 us; skew 500 ppm
I20260927 05:15:57.081195 14329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:57 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:57.084424 14329 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:15:57.085724 14329 webserver.cc:460] Webserver started at http://127.0.97.1:39185/ using document root <none> and password file <none>
I20260927 05:15:57.086297 14329 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:57.086406 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:57.086701 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:57.089440 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "654d35062dff4f8897f9db0ef9681c98"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "48f124b58389d784f47fca8cba62cda2"
server_key_iv: "d90a67b0bc1bda0c5605c4f7df48701d"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.090186 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "654d35062dff4f8897f9db0ef9681c98"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "48f124b58389d784f47fca8cba62cda2"
server_key_iv: "d90a67b0bc1bda0c5605c4f7df48701d"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.094872 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:15:57.098044 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.099319 14329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:57.099478 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "654d35062dff4f8897f9db0ef9681c98"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "48f124b58389d784f47fca8cba62cda2"
server_key_iv: "d90a67b0bc1bda0c5605c4f7df48701d"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.099696 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:57.117848 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:57.121866 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:57.122102 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:57.122915 14329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:57.124142 14329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:57.124212 14329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.124293 14329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:57.124336 14329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.156381 14329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43427
I20260927 05:15:57.156461 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43427 every 8 connection(s)
I20260927 05:15:57.157590 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:15:57.166047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14329
I20260927 05:15:57.166258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:15:57.166863 388 external_mini_cluster.cc:1428] Setting key 62db0e9fa9a3fdaede55e0a69048e788
Sep 27 05:15:57 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:57.176661 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:57.160076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46737 (local address 127.0.97.62:40991)
0927 05:15:57.160504 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:57.160509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:57.161428 (+ 919us) server_negotiation.cc:407] Connection header received
0927 05:15:57.163228 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:57.163232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:57.163296 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:57.163392 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:57.164714 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.165431 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.167043 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.167456 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.167598 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:57.171136 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:57.171141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:57.171144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:57.171175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:57.173703 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:57.174380 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:57.174386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:57.174392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:57.174518 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:57.174934 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:57.174940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:57.174943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:57.175274 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:57.175431 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:57.176071 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 05:15:57.176302 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260927 05:15:57.177541 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:57.160609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:46737)
0927 05:15:57.161272 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:57.161310 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:57.162907 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:57.163582 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:57.163589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:57.164044 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:57.164571 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:57.164586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.165554 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.165560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:57.166874 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:57.166885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.167682 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.167689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:57.167917 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:57.168632 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:57.168670 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:57.170881 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:57.173840 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:57.173849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:57.173865 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:57.174265 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:57.174633 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:57.174638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:57.174642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:57.174823 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:57.175399 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:57.175406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:57.175725 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 05:15:57.175985 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 05:15:57.176478 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":255,"threads_started":1}
I20260927 05:15:57.179124 14457 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:15:57.179414 14457 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:57.180086 14457 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:15:57.182104 14268 ts_manager.cc:194] Registered new tserver with Master: 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427)
I20260927 05:15:57.183588 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46737
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:15:57.216539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:57.529381 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:57.529682 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:57.529845 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:57.537442 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:57.537535 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:57.537644 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:15:57.546900 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:57.548031 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:57.549208 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:57.561466 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:57.561437 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:57.561403 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:57.562462 14463 server_base.cc:1034] running on GCE node
I20260927 05:15:57.563223 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:57.564095 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:57.565317 14463 hybrid_clock.cc:648] HybridClock initialized: now 1790486157565266 us; error 42 us; skew 500 ppm
I20260927 05:15:57.565645 14463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:57 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:57.568604 14463 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:15:57.569883 14463 webserver.cc:460] Webserver started at http://127.0.97.2:43553/ using document root <none> and password file <none>
I20260927 05:15:57.570503 14463 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:57.570657 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:57.570942 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:57.573805 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f9a203ace134e6983097838089ed521"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "95b8cc677c11e930b0dcd77381b29654"
server_key_iv: "d4f4a98af36d4ed7df3a4c8198534552"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.574581 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f9a203ace134e6983097838089ed521"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "95b8cc677c11e930b0dcd77381b29654"
server_key_iv: "d4f4a98af36d4ed7df3a4c8198534552"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.579492 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 05:15:57.582702 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.583972 14463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:57.584146 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "4f9a203ace134e6983097838089ed521"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "95b8cc677c11e930b0dcd77381b29654"
server_key_iv: "d4f4a98af36d4ed7df3a4c8198534552"
server_key_version: "encryptionkey@0"
I20260927 05:15:57.584340 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:57.615274 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:57.618928 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:57.619228 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:57.620152 14463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:57.621436 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:57.621538 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.621686 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:57.621767 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:57.654495 14463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33937
I20260927 05:15:57.654574 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33937 every 8 connection(s)
I20260927 05:15:57.655768 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:15:57.656797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14463
I20260927 05:15:57.656996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:15:57.657580 388 external_mini_cluster.cc:1428] Setting key bf92e64d563bc31a9af6fd59ab98bc7e
Sep 27 05:15:57 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:57.682754 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:57.658558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48657 (local address 127.0.97.62:40991)
0927 05:15:57.658778 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:57.658787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:57.660368 (+ 1581us) server_negotiation.cc:407] Connection header received
0927 05:15:57.662115 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:57.662123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:57.662217 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:57.662330 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:57.664409 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.665639 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.667940 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.668718 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.668864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:57.675881 (+ 7017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:57.675891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:57.675900 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:57.675946 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:57.679585 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:57.680501 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:57.680506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:57.680508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:57.680633 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:57.681115 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:57.681118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:57.681120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:57.681358 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:57.681467 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:57.682309 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 05:15:57.682488 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:15:57.684286 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:57.659139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:48657)
0927 05:15:57.660190 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:57.660233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:57.661856 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:57.662657 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:57.662671 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:57.663443 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:57.664243 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:57.664270 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.665822 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.665830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:57.667733 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:57.667746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:57.669015 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:57.669023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:57.669404 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:57.670137 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:57.670163 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:57.675612 (+ 5449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:57.679754 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:57.679766 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:57.679784 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:57.680377 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:57.680755 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:57.680763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:57.680768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:57.681006 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:57.681500 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:57.681510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:57.681945 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 05:15:57.682224 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 05:15:57.682816 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":570,"threads_started":1}
I20260927 05:15:57.685760 14591 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:15:57.686072 14591 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:57.686794 14591 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:15:57.688313 14268 ts_manager.cc:194] Registered new tserver with Master: 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937)
I20260927 05:15:57.689188 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48657
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:15:57.717753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:58.028307 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:58.028561 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:58.028661 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:58.035956 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:58.036051 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:58.036159 14596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:15:58.045344 14596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:58.046445 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:58.047765 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:58.060086 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:58.060192 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:58.060130 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:58.060827 14596 server_base.cc:1034] running on GCE node
I20260927 05:15:58.061424 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:58.062271 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:58.063496 14596 hybrid_clock.cc:648] HybridClock initialized: now 1790486158063452 us; error 53 us; skew 500 ppm
I20260927 05:15:58.063808 14596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:58 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:58.066918 14596 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:15:58.068230 14596 webserver.cc:460] Webserver started at http://127.0.97.3:45897/ using document root <none> and password file <none>
I20260927 05:15:58.068819 14596 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:58.068917 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:58.069141 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:15:58.071866 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "c6b9b5820c924c059627fbab87d50c34"
format_stamp: "Formatted at 2026-09-27 05:15:58 on dist-test-slave-1t6p"
server_key: "91eda73b8546cbfe673ad320727ed56f"
server_key_iv: "2398e11ac30f5227065399a617ef6273"
server_key_version: "encryptionkey@0"
I20260927 05:15:58.072711 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c6b9b5820c924c059627fbab87d50c34"
format_stamp: "Formatted at 2026-09-27 05:15:58 on dist-test-slave-1t6p"
server_key: "91eda73b8546cbfe673ad320727ed56f"
server_key_iv: "2398e11ac30f5227065399a617ef6273"
server_key_version: "encryptionkey@0"
I20260927 05:15:58.077644 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:15:58.080956 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:58.082196 14596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:15:58.082368 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "c6b9b5820c924c059627fbab87d50c34"
format_stamp: "Formatted at 2026-09-27 05:15:58 on dist-test-slave-1t6p"
server_key: "91eda73b8546cbfe673ad320727ed56f"
server_key_iv: "2398e11ac30f5227065399a617ef6273"
server_key_version: "encryptionkey@0"
I20260927 05:15:58.082552 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:58.095480 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:58.099599 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:58.099879 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:58.100750 14596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:15:58.101975 14596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:15:58.102046 14596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:58.102211 14596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:15:58.102267 14596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:58.133721 14596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42199
I20260927 05:15:58.133785 14723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42199 every 8 connection(s)
I20260927 05:15:58.135054 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:15:58.136638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14596
I20260927 05:15:58.136837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:15:58.137320 388 external_mini_cluster.cc:1428] Setting key bbc78d11af6ce1d44d10f90a5854ff45
Sep 27 05:15:58 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:15:58.154211 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.137825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42091 (local address 127.0.97.62:40991)
0927 05:15:58.137990 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.137995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.139488 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 05:15:58.141405 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.141410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.141489 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.141602 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:58.142972 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.143762 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.145224 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.145623 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.145753 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.148981 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:58.148986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:58.148989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:58.149025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:58.151460 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.152326 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.152332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.152335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.152491 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.152891 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.152894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.152896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.153138 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:58.153256 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.153822 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.153971 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:15:58.155109 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.138439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:42091)
0927 05:15:58.139348 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.139402 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.141164 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.141827 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.141834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.142281 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:15:58.142819 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.142835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.143910 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.143915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.145082 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.145093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.145832 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.145842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.146096 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.146690 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:15:58.146710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:15:58.148799 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:15:58.151590 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:58.151600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:58.151614 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:58.152171 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:58.152610 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:15:58.152615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:15:58.152619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:15:58.152799 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:15:58.153253 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:15:58.153259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:15:58.153523 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.153739 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.154064 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":329,"threads_started":1}
I20260927 05:15:58.156332 14724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:15:58.156646 14724 heartbeater.cc:464] Registering TS with master...
I20260927 05:15:58.157313 14724 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:15:58.158623 14268 ts_manager.cc:194] Registered new tserver with Master: c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199)
I20260927 05:15:58.159463 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42091
I20260927 05:15:58.164364 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:15:58.179996 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.168384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.0.97.62:40991)
0927 05:15:58.168550 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.168557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.168973 (+ 416us) server_negotiation.cc:407] Connection header received
0927 05:15:58.169177 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.169181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.169238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.169336 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:58.170722 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.171473 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.172736 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.173131 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.173226 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.175417 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:58.175422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:58.175425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:58.175458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:58.177673 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.178231 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.178235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.178237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.178365 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.178695 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.178698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.178700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.178985 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:58.179222 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.179547 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.179685 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:15:58.186329 14457 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:15:58.183395 14268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:15:58.191449 14268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:15:58.223718 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.203372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.97.3:42199)
0927 05:15:58.211566 (+ 8194us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.211578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.216097 (+ 4519us) server_negotiation.cc:407] Connection header received
0927 05:15:58.216384 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.216392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.216578 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.216716 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:58.217705 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.219139 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.221641 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.222520 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.222666 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.222898 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.223208 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.223360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8002,"thread_start_us":7853,"threads_started":1}
I20260927 05:15:58.223762 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.202657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42199 (local address 127.0.0.1:41456)
0927 05:15:58.216001 (+ 13344us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.216017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.216210 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.216991 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.216994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.217009 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:58.217518 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.217532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.219295 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.219301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.221511 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.221520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.222718 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.222724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.222903 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.222962 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.223159 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.223360 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13238,"mutex_wait_us":24,"thread_start_us":9701,"threads_started":1}
I20260927 05:15:58.224246 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.202078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.0.1:48368)
0927 05:15:58.212842 (+ 10764us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.212901 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.213117 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.216440 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.216447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.216481 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:58.216926 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.216940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.218586 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.218591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.221217 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.221231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.223215 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.223222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.223435 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.223553 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.223652 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.223811 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10454,"mutex_wait_us":127,"thread_start_us":13380,"threads_started":1}
I20260927 05:15:58.224979 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.202337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.97.1:43427)
0927 05:15:58.215480 (+ 13143us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.215491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.215524 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:15:58.215641 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.215651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.215942 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.216130 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:58.217091 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.218450 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.221374 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.222707 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.222812 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.223049 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.224503 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.224675 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12933,"thread_start_us":12758,"threads_started":1}
I20260927 05:15:58.225559 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.202728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.0.97.2:33937)
0927 05:15:58.214691 (+ 11963us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.214704 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.216081 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 05:15:58.216183 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.216192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.216503 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.216702 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:58.217715 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.218770 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.221204 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.222673 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.222810 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.222985 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.225189 (+ 2204us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.225328 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11760,"thread_start_us":11994,"threads_started":1}
I20260927 05:15:58.225020 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.202227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33937 (local address 127.0.0.1:58608)
0927 05:15:58.215780 (+ 13553us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.215804 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.215944 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.216970 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.216975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.216999 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:58.217467 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.217484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.218917 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.218922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.221047 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.221059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.223215 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.223222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.223451 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.224477 (+ 1026us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.224559 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.224675 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13371,"mutex_wait_us":148,"thread_start_us":13533,"threads_started":1}
I20260927 05:15:58.227849 14659 tablet_service.cc:1467] Processing CreateTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a (DEFAULT_TABLE table=test-table [id=cfa9bd8243404eceb7a13c4938cc802c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:58.228216 14392 tablet_service.cc:1467] Processing CreateTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a (DEFAULT_TABLE table=test-table [id=cfa9bd8243404eceb7a13c4938cc802c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:58.229647 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bfd46bd2534b70b79b78cd95dad11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:58.229216 14526 tablet_service.cc:1467] Processing CreateTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a (DEFAULT_TABLE table=test-table [id=cfa9bd8243404eceb7a13c4938cc802c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:15:58.229822 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bfd46bd2534b70b79b78cd95dad11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:58.230386 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bfd46bd2534b70b79b78cd95dad11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:15:58.240521 14742 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Bootstrap starting.
I20260927 05:15:58.243645 14743 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Bootstrap starting.
I20260927 05:15:58.244083 14742 tablet_bootstrap.cc:654] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:58.245303 14742 log.cc:824] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:58.246675 14744 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Bootstrap starting.
I20260927 05:15:58.247025 14743 tablet_bootstrap.cc:654] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:58.248087 14742 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: No bootstrap required, opened a new log
I20260927 05:15:58.248176 14743 log.cc:824] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:58.248265 14742 ts_tablet_manager.cc:1397] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:15:58.249749 14744 tablet_bootstrap.cc:654] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Neither blocks nor log segments found. Creating new log.
I20260927 05:15:58.250898 14744 log.cc:824] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:58.251499 14743 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: No bootstrap required, opened a new log
I20260927 05:15:58.251724 14743 ts_tablet_manager.cc:1397] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 05:15:58.253618 14744 tablet_bootstrap.cc:492] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: No bootstrap required, opened a new log
I20260927 05:15:58.253890 14744 ts_tablet_manager.cc:1397] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 05:15:58.256462 14742 raft_consensus.cc:348] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.256811 14742 raft_consensus.cc:374] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:58.256950 14742 raft_consensus.cc:729] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654d35062dff4f8897f9db0ef9681c98, State: Initialized, Role: FOLLOWER
I20260927 05:15:58.257513 14742 consensus_queue.cc:260] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.258991 14742 ts_tablet_manager.cc:1428] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260927 05:15:58.261994 14743 raft_consensus.cc:348] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.262396 14743 raft_consensus.cc:374] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:58.262488 14743 raft_consensus.cc:729] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9b5820c924c059627fbab87d50c34, State: Initialized, Role: FOLLOWER
I20260927 05:15:58.263116 14743 consensus_queue.cc:260] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.264189 14724 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:15:58.264761 14743 ts_tablet_manager.cc:1428] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 05:15:58.265380 14744 raft_consensus.cc:348] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.265793 14744 raft_consensus.cc:374] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:15:58.265901 14744 raft_consensus.cc:729] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9a203ace134e6983097838089ed521, State: Initialized, Role: FOLLOWER
I20260927 05:15:58.266461 14744 consensus_queue.cc:260] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.267594 14591 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:15:58.268182 14744 ts_tablet_manager.cc:1428] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260927 05:15:58.386898 14725 tablet.cc:2369] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:15:58.407742 14592 tablet.cc:2369] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:15:58.410087 14458 tablet.cc:2369] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:15:58.568826 14749 raft_consensus.cc:482] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:15:58.569100 14749 raft_consensus.cc:504] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.570808 14749 leader_election.cc:288] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427), 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937)
I20260927 05:15:58.577292 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.571217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40841 (local address 127.0.97.1:43427)
0927 05:15:58.571419 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.571425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.571447 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:15:58.571741 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.571747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.571833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.571935 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:58.572363 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.573492 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.575349 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.576450 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.576572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.576643 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.577026 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.577117 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:15:58.577395 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.570999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.97.3:40841)
0927 05:15:58.571328 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.571347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.571481 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.571962 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.571965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.571986 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:58.572199 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.572225 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.573645 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.573650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.575202 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.575213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.576616 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.576622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.576822 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.576871 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.576969 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.577117 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 05:15:58.578300 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.571223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33937 (local address 127.0.97.3:46919)
0927 05:15:58.571971 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 05:15:58.571989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:15:58.572158 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:15:58.572490 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:15:58.572497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:15:58.572521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:15:58.572880 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.572891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.574252 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.574256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.576015 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:15:58.576030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.577463 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.577472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:15:58.577745 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.577789 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 05:15:58.577902 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 05:15:58.578028 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"spinlock_wait_cycles":19712,"thread_start_us":381,"threads_started":1}
I20260927 05:15:58.578296 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.571305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46919 (local address 127.0.97.2:33937)
0927 05:15:58.571612 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.571619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.572085 (+ 466us) server_negotiation.cc:407] Connection header received
0927 05:15:58.572301 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.572308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.572384 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.572447 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:15:58.573070 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.574075 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.576170 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.577239 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.577362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.577553 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.577933 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.578037 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 05:15:58.578572 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bfd46bd2534b70b79b78cd95dad11a" candidate_uuid: "c6b9b5820c924c059627fbab87d50c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654d35062dff4f8897f9db0ef9681c98" is_pre_election: true
I20260927 05:15:58.579144 14412 raft_consensus.cc:2454] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b9b5820c924c059627fbab87d50c34 in term 0.
I20260927 05:15:58.579304 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bfd46bd2534b70b79b78cd95dad11a" candidate_uuid: "c6b9b5820c924c059627fbab87d50c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f9a203ace134e6983097838089ed521" is_pre_election: true
I20260927 05:15:58.579762 14615 leader_election.cc:302] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 654d35062dff4f8897f9db0ef9681c98, c6b9b5820c924c059627fbab87d50c34; no voters:
I20260927 05:15:58.579838 14546 raft_consensus.cc:2454] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b9b5820c924c059627fbab87d50c34 in term 0.
I20260927 05:15:58.580160 14749 raft_consensus.cc:2781] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:15:58.580325 14749 raft_consensus.cc:482] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:15:58.580487 14749 raft_consensus.cc:3037] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:58.582546 14749 raft_consensus.cc:504] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.583173 14749 leader_election.cc:288] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [CANDIDATE]: Term 1 election: Requested vote from peers 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427), 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937)
I20260927 05:15:58.583464 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bfd46bd2534b70b79b78cd95dad11a" candidate_uuid: "c6b9b5820c924c059627fbab87d50c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654d35062dff4f8897f9db0ef9681c98"
I20260927 05:15:58.583648 14412 raft_consensus.cc:3037] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:58.583664 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bfd46bd2534b70b79b78cd95dad11a" candidate_uuid: "c6b9b5820c924c059627fbab87d50c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f9a203ace134e6983097838089ed521"
I20260927 05:15:58.583853 14546 raft_consensus.cc:3037] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:15:58.585876 14412 raft_consensus.cc:2454] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b9b5820c924c059627fbab87d50c34 in term 1.
I20260927 05:15:58.586220 14546 raft_consensus.cc:2454] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b9b5820c924c059627fbab87d50c34 in term 1.
I20260927 05:15:58.586272 14615 leader_election.cc:302] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 654d35062dff4f8897f9db0ef9681c98, c6b9b5820c924c059627fbab87d50c34; no voters:
I20260927 05:15:58.586553 14749 raft_consensus.cc:2781] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:15:58.586946 14749 raft_consensus.cc:686] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 1 LEADER]: Becoming Leader. State: Replica: c6b9b5820c924c059627fbab87d50c34, State: Running, Role: LEADER
I20260927 05:15:58.587433 14749 consensus_queue.cc:237] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:15:58.592159 14268 catalog_manager.cc:5495] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b9b5820c924c059627fbab87d50c34 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c6b9b5820c924c059627fbab87d50c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } health_report { overall_health: HEALTHY } } }
I20260927 05:15:58.673316 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.668404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.97.3:42199)
0927 05:15:58.668572 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.668576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.668596 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:15:58.668774 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.668777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.668872 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.669013 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:15:58.669370 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.670072 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.671532 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.671906 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.671998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.672419 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:15:58.672861 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:15:58.672974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.673059 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.673119 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:15:58.680902 14546 raft_consensus.cc:1261] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Refusing update from remote peer c6b9b5820c924c059627fbab87d50c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:58.680950 14412 raft_consensus.cc:1261] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Refusing update from remote peer c6b9b5820c924c059627fbab87d50c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:15:58.681813 14752 consensus_queue.cc:1035] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:58.682127 14749 consensus_queue.cc:1035] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:15:58.698643 14755 mvcc.cc:205] Tried to move back new op lower bound from 7333831305947537408 to 7333831305578024960. Current Snapshot: MvccSnapshot[applied={T|T < 7333831305947537408}]
I20260927 05:15:58.701020 14757 mvcc.cc:205] Tried to move back new op lower bound from 7333831305947537408 to 7333831305578024960. Current Snapshot: MvccSnapshot[applied={T|T < 7333831305947537408 or (T in {7333831305947537408})}]
I20260927 05:15:58.710249 14267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:15:58.710593 14267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:15:58.714003 14267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Sending DeleteTablet for 3 replicas of tablet 87bfd46bd2534b70b79b78cd95dad11a
I20260927 05:15:58.715045 14526 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58608
I20260927 05:15:58.715055 14392 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48368
I20260927 05:15:58.715545 14659 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bfd46bd2534b70b79b78cd95dad11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:15:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41456
I20260927 05:15:58.715780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14329
I20260927 05:15:58.732766 14764 tablet_replica.cc:330] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: stopping tablet replica
I20260927 05:15:58.733349 14764 raft_consensus.cc:2229] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:15:58.733927 14764 raft_consensus.cc:2258] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:58.735236 14766 tablet_replica.cc:330] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: stopping tablet replica
I20260927 05:15:58.735466 14765 tablet_replica.cc:330] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: stopping tablet replica
I20260927 05:15:58.735610 14766 raft_consensus.cc:2229] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:15:58.735864 14765 raft_consensus.cc:2229] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:15:58.736253 14764 ts_tablet_manager.cc:1905] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:58.736348 14766 raft_consensus.cc:2258] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:58.736474 14765 raft_consensus.cc:2258] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:15:58.738508 14765 ts_tablet_manager.cc:1905] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:58.738514 14766 ts_tablet_manager.cc:1905] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:15:58.743289 14764 ts_tablet_manager.cc:1918] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:58.743464 14764 log.cc:1196] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/87bfd46bd2534b70b79b78cd95dad11a
I20260927 05:15:58.744029 14764 ts_tablet_manager.cc:1939] T 87bfd46bd2534b70b79b78cd95dad11a P 654d35062dff4f8897f9db0ef9681c98: Deleting consensus metadata
Sep 27 05:15:58 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:15:58.745584 14765 ts_tablet_manager.cc:1918] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:58.745761 14765 log.cc:1196] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/87bfd46bd2534b70b79b78cd95dad11a
I20260927 05:15:58.745823 14255 catalog_manager.cc:4841] TS 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427): tablet 87bfd46bd2534b70b79b78cd95dad11a (table test-table [id=cfa9bd8243404eceb7a13c4938cc802c]) successfully deleted
I20260927 05:15:58.746294 14765 ts_tablet_manager.cc:1939] T 87bfd46bd2534b70b79b78cd95dad11a P 4f9a203ace134e6983097838089ed521: Deleting consensus metadata
I20260927 05:15:58.747851 14254 catalog_manager.cc:4841] TS 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937): tablet 87bfd46bd2534b70b79b78cd95dad11a (table test-table [id=cfa9bd8243404eceb7a13c4938cc802c]) successfully deleted
I20260927 05:15:58.751263 14766 ts_tablet_manager.cc:1918] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:15:58.751420 14766 log.cc:1196] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/87bfd46bd2534b70b79b78cd95dad11a
I20260927 05:15:58.751977 14766 ts_tablet_manager.cc:1939] T 87bfd46bd2534b70b79b78cd95dad11a P c6b9b5820c924c059627fbab87d50c34: Deleting consensus metadata
I20260927 05:15:58.753362 14252 catalog_manager.cc:4841] TS c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199): tablet 87bfd46bd2534b70b79b78cd95dad11a (table test-table [id=cfa9bd8243404eceb7a13c4938cc802c]) successfully deleted
I20260927 05:15:58.755106 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.716849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.97.1:43427)
0927 05:15:58.717014 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.717022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.724357 (+ 7335us) server_negotiation.cc:407] Connection header received
0927 05:15:58.725349 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.725359 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.725458 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.725562 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:58.730269 (+ 4707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.731865 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.739135 (+ 7270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.739769 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.739863 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.748310 (+ 8447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:58.748324 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:58.748340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:58.748382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:58.752246 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.753094 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.753102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.753106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.753273 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.753698 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.753702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.753704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.754005 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:58.754252 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.754602 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.754767 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:15:58.755859 14452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:58.844280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14329
I20260927 05:15:58.868957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14463
Sep 27 05:15:58 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:15:58.883834 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.869759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.0.97.2:33937)
0927 05:15:58.869933 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.869941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.869964 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:15:58.870183 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.870189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.870267 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.870372 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:58.871660 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.872345 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.873620 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.874034 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.874161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:58.878280 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:58.878294 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:58.878308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:58.878341 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:58.881078 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.881751 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.881759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.881761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.881887 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:58.882257 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:58.882264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:58.882267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:58.882619 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:58.882919 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:58.883289 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 05:15:58.883506 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:15:58.884522 14586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:58.973806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14463
I20260927 05:15:58.991139 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14596
Sep 27 05:15:58 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:15:59.006686 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:15:58.991866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.97.3:42199)
0927 05:15:58.992063 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:15:58.992073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:15:58.992176 (+ 103us) server_negotiation.cc:407] Connection header received
0927 05:15:58.992400 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:15:58.992405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:15:58.992477 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:15:58.992577 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:15:58.993893 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.994717 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.996020 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:15:58.996513 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:15:58.996645 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:15:59.000870 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:15:59.000881 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:15:59.000894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:15:59.000930 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:15:59.003892 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:59.004574 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:59.004583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:59.004593 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:59.004752 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:15:59.005130 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:15:59.005136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:15:59.005138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:15:59.005493 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:15:59.005827 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 05:15:59.006116 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 05:15:59.006332 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:15:59.007434 14719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:59.126607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14596
I20260927 05:15:59.145807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14236
I20260927 05:15:59.146879 14298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:15:59.256412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14236
I20260927 05:15:59.278215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--webserver_interface=127.0.97.62
--webserver_port=44291
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:15:59.593161 14772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:15:59.593420 14772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:15:59.593592 14772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:15:59.600488 14772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:15:59.600587 14772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:15:59.600694 14772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:15:59.600800 14772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:15:59.600881 14772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:15:59.611716 14772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:15:59.612962 14772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:15:59.614248 14772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:15:59.636112 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:15:59.636169 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:59.636816 14772 server_base.cc:1034] running on GCE node
W20260927 05:15:59.636396 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:15:59.637873 14772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:15:59.639348 14772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:15:59.640553 14772 hybrid_clock.cc:648] HybridClock initialized: now 1790486159640499 us; error 34 us; skew 500 ppm
I20260927 05:15:59.640976 14772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:15:59 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:15:59.644366 14772 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:15:59.646076 14772 webserver.cc:460] Webserver started at http://127.0.97.62:44291/ using document root <none> and password file <none>
I20260927 05:15:59.646819 14772 fs_manager.cc:359] Metadata directory not provided
I20260927 05:15:59.646976 14772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:15:59.652545 14772 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:15:59.655419 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:15:59.656709 14772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:15:59.656888 14772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "012b4dc648a14bb9a38600ecd433dcde"
format_stamp: "Formatted at 2026-09-27 05:15:56 on dist-test-slave-1t6p"
server_key: "388fa87410a643f883bba383210b31e2"
server_key_iv: "4ae817edaf3488df03bd701330ee13d4"
server_key_version: "encryptionkey@0"
I20260927 05:15:59.657918 14772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:15:59.673769 14772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:15:59.677973 14772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:15:59.678263 14772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:15:59.697129 14772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40991
I20260927 05:15:59.697150 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40991 every 8 connection(s)
I20260927 05:15:59.698650 14772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:15:59.705261 14840 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:15:59.707904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14772
I20260927 05:15:59.708565 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap starting.
I20260927 05:15:59.708760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43427
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=39185
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:15:59.727731 14840 log.cc:824] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Log is configured to *not* fsync() on all Append() calls
I20260927 05:15:59.739193 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:15:59.739704 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap complete.
I20260927 05:15:59.748672 14840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:59.749487 14840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Initialized, Role: FOLLOWER
I20260927 05:15:59.750038 14840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:59.750298 14840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:15:59.750424 14840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:15:59.750607 14840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:15:59.754261 14840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:59.754735 14840 leader_election.cc:302] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012b4dc648a14bb9a38600ecd433dcde; no voters:
I20260927 05:15:59.755385 14840 leader_election.cc:288] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:15:59.755505 14844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:15:59.756915 14844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 LEADER]: Becoming Leader. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Running, Role: LEADER
I20260927 05:15:59.757490 14844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:15:59.757694 14840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:15:59.760106 14845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:15:59.760560 14845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:59.760514 14846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012b4dc648a14bb9a38600ecd433dcde. Latest consensus state: current_term: 2 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:15:59.760860 14846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:15:59.766347 14849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:15:59.775776 14849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:15:59.777585 14849 catalog_manager.cc:1259] Loaded cluster ID: 0de358c8977146628782bb2d31ccb74c
I20260927 05:15:59.777752 14849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:15:59.781670 14849 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 05:15:59.799994 14849 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 05:15:59.806547 14849 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:15:59.807862 14849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Loaded TSK: 0
I20260927 05:15:59.808938 14849 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:16:00.060776 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:00.061070 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:00.061229 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:00.069083 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:00.069221 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:00.069401 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:16:00.078681 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39185
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:00.079911 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:00.081180 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:00.093588 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.093623 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.093842 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:00.095490 14842 server_base.cc:1034] running on GCE node
I20260927 05:16:00.096127 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:00.097036 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:00.098265 14842 hybrid_clock.cc:648] HybridClock initialized: now 1790486160098208 us; error 52 us; skew 500 ppm
I20260927 05:16:00.098599 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:00 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:00.101723 14842 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:16:00.103191 14842 webserver.cc:460] Webserver started at http://127.0.97.1:39185/ using document root <none> and password file <none>
I20260927 05:16:00.103858 14842 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:00.103994 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:00.109197 14842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 05:16:00.111881 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.113332 14842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:00.113500 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "654d35062dff4f8897f9db0ef9681c98"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "48f124b58389d784f47fca8cba62cda2"
server_key_iv: "d90a67b0bc1bda0c5605c4f7df48701d"
server_key_version: "encryptionkey@0"
I20260927 05:16:00.114423 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:00.137066 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:00.141325 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:00.141607 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:00.142556 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:00.144017 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:00.144138 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.144269 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:00.144361 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.177112 14842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43427
I20260927 05:16:00.177201 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43427 every 8 connection(s)
I20260927 05:16:00.178508 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:16:00.180188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14842
I20260927 05:16:00.180737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33937
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=43553
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:16:00 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:00.207472 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:00.185161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:48989)
0927 05:16:00.186059 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:00.186118 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:00.188111 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:00.189584 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:00.189591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:00.190114 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:00.190643 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:00.190661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.193021 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.193025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:00.194204 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:00.194215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.195727 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.195733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:00.195912 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:00.196603 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:00.196625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:00.198800 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:00.203613 (+ 4813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:00.203620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:00.203630 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:00.204013 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:00.204508 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:00.204512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:00.204514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:00.204675 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:00.205590 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:00.205595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:00.205849 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 05:16:00.206058 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 05:16:00.206410 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":352,"threads_started":1}
I20260927 05:16:00.207772 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:00.185023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48989 (local address 127.0.97.62:40991)
0927 05:16:00.185730 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:00.185746 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:00.186258 (+ 512us) server_negotiation.cc:407] Connection header received
0927 05:16:00.188682 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:00.188702 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:00.189116 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:00.189476 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:00.190873 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.192876 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.194375 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.195517 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.195710 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:00.199108 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:00.199122 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:00.199145 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:00.199199 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:00.203476 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:00.204152 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:00.204161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:00.204171 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:00.204386 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:00.204793 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:00.204799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:00.204805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:00.205441 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:00.205641 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:00.206281 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 05:16:00.206677 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
I20260927 05:16:00.210392 14990 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:16:00.210690 14990 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:00.211416 14990 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:16:00.213538 14805 ts_manager.cc:194] Registered new tserver with Master: 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427)
I20260927 05:16:00.214977 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48989
W20260927 05:16:00.501036 14994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:00.501334 14994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:00.501505 14994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:00.508565 14994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:00.508708 14994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:00.508896 14994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:16:00.518500 14994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43553
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:00.519726 14994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:00.520962 14994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:00.533177 14999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.533233 15002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.533237 15000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:00.534267 14994 server_base.cc:1034] running on GCE node
I20260927 05:16:00.534843 14994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:00.535768 14994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:00.536993 14994 hybrid_clock.cc:648] HybridClock initialized: now 1790486160536962 us; error 44 us; skew 500 ppm
I20260927 05:16:00.537295 14994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:00 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:00.540326 14994 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:16:00.541712 14994 webserver.cc:460] Webserver started at http://127.0.97.2:43553/ using document root <none> and password file <none>
I20260927 05:16:00.542351 14994 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:00.542511 14994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:00.547528 14994 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:16:00.550097 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.551595 14994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:00.551764 14994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "4f9a203ace134e6983097838089ed521"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "95b8cc677c11e930b0dcd77381b29654"
server_key_iv: "d4f4a98af36d4ed7df3a4c8198534552"
server_key_version: "encryptionkey@0"
I20260927 05:16:00.552709 14994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:00.569530 14994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:00.573653 14994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:00.573922 14994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:00.574791 14994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:00.576052 14994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:00.576174 14994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.576304 14994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:00.576393 14994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.609349 14994 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33937
I20260927 05:16:00.609434 15121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33937 every 8 connection(s)
I20260927 05:16:00.611155 14994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:16:00.613592 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 14994
I20260927 05:16:00.614172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42199
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=45897
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:16:00 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:00.636595 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:00.615685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38101 (local address 127.0.97.62:40991)
0927 05:16:00.615872 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:00.615878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:00.617238 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 05:16:00.620209 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:00.620214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:00.620284 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:00.620371 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:00.622185 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.623378 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.625072 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.625868 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.626004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:00.629646 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:00.629655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:00.629662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:00.629709 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:00.633427 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:00.634085 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:00.634093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:00.634099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:00.634258 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:00.634654 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:00.634662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:00.634668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:00.635085 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:00.635243 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:00.635860 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 05:16:00.636115 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:16:00.637102 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:00.616217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:38101)
0927 05:16:00.617051 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:00.617098 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:00.619901 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:00.620637 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:00.620651 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:00.621444 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:00.621990 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:00.622006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.623700 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.623704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:00.624912 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:00.624924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:00.626080 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:00.626085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:00.626289 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:00.627072 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:00.627103 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:00.629427 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:00.633567 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:00.633574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:00.633585 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:00.633963 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:00.634371 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:00.634374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:00.634377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:00.634540 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:00.635220 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:00.635226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:00.635501 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 05:16:00.635713 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 05:16:00.636042 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":323,"threads_started":1}
I20260927 05:16:00.638561 15122 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:16:00.638895 15122 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:00.639846 15122 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:16:00.641286 14805 ts_manager.cc:194] Registered new tserver with Master: 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937)
I20260927 05:16:00.642158 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38101
W20260927 05:16:00.939291 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:00.939592 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:00.939760 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:00.946671 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:00.946825 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:00.947037 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:16:00.956224 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45897
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:00.957464 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:00.958833 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:00.971935 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.971902 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:00.972172 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:00.972622 15125 server_base.cc:1034] running on GCE node
I20260927 05:16:00.973184 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:00.974047 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:00.975271 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790486160975245 us; error 46 us; skew 500 ppm
I20260927 05:16:00.975570 15125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:00 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:00.978519 15125 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:16:00.979878 15125 webserver.cc:460] Webserver started at http://127.0.97.3:45897/ using document root <none> and password file <none>
I20260927 05:16:00.980456 15125 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:00.980561 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:00.985458 15125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:16:00.988019 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:00.989259 15125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:16:00.989450 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "c6b9b5820c924c059627fbab87d50c34"
format_stamp: "Formatted at 2026-09-27 05:15:58 on dist-test-slave-1t6p"
server_key: "91eda73b8546cbfe673ad320727ed56f"
server_key_iv: "2398e11ac30f5227065399a617ef6273"
server_key_version: "encryptionkey@0"
I20260927 05:16:00.990406 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:01.029601 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:01.033766 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:01.034049 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:01.035100 15125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:01.036410 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:01.036504 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:01.036679 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:01.036774 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:01.070556 15125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42199
I20260927 05:16:01.070596 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42199 every 8 connection(s)
I20260927 05:16:01.072006 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:16:01.078542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15125
Sep 27 05:16:01 dist-test-slave-1t6p krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:01.094214 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.074447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53157 (local address 127.0.97.62:40991)
0927 05:16:01.074629 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.074634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.075846 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 05:16:01.077609 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.077615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.077708 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.077823 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:01.079721 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.080770 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.082432 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.082860 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.082950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.087161 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:01.087166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:01.087169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:01.087198 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:01.090966 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.091681 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.091687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.091690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.091832 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.092304 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.092307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.092309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.092686 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:01.092818 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.093601 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.093853 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"spinlock_wait_cycles":2688}
I20260927 05:16:01.094650 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.079507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.97.62:40991)
0927 05:16:01.080431 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.080435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.080450 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:16:01.080487 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.080490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.080549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.080636 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:01.082116 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.083051 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.084528 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.085189 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.085325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.088274 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:01.088285 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:01.088291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:01.088340 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:01.091723 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.092516 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.092521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.092524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.092677 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.093063 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.093066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.093068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.093350 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:01.093643 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.093964 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.094208 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":700,"threads_started":1}
I20260927 05:16:01.095301 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.074916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:53157)
0927 05:16:01.075690 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.075731 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.077354 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.078050 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.078057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.078819 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:01.079552 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.079570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.080932 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.080936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.082289 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.082305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.083112 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.083123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.083352 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.084029 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:01.084050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:01.086905 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:01.091133 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:01.091140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:01.091150 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:01.091545 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:01.091944 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:01.091951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:01.091957 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:01.092168 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:01.092812 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:01.092819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:01.093213 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.093448 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.093919 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":275,"threads_started":1}
I20260927 05:16:01.096659 15253 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40991
I20260927 05:16:01.096956 15253 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:01.097658 15253 heartbeater.cc:511] Master 127.0.97.62:40991 requested a full tablet report, sending...
I20260927 05:16:01.098876 14805 ts_manager.cc:194] Registered new tserver with Master: c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199)
I20260927 05:16:01.099673 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53157
I20260927 05:16:01.109434 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:16:01.116092 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.110395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.0.97.62:40991)
0927 05:16:01.110561 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.110566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.111056 (+ 490us) server_negotiation.cc:407] Connection header received
0927 05:16:01.111265 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.111269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.111388 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.111496 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:16:01.111900 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.112533 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.113982 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.114396 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.114482 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.115109 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:16:01.115654 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:16:01.115757 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.115873 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.115929 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:16:01.117008 14805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:16:01.124105 14805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:16:01.145409 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.135567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33937 (local address 127.0.0.1:58632)
0927 05:16:01.136562 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.136614 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.136799 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.137941 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.137947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.137978 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:01.138452 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.138465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.140418 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.140423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.143084 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.143100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.144412 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.144418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.144603 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.144647 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.144785 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.144953 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":28,"thread_start_us":2650,"threads_started":1}
I20260927 05:16:01.145473 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.135714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.97.2:33937)
0927 05:16:01.137346 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.137354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.137387 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:16:01.137476 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.137482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.137727 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.137932 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:01.138686 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.140274 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.143251 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.144160 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.144325 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.144536 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.144834 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.145167 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":1639,"threads_started":1}
I20260927 05:16:01.148428 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.138357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.97.3:42199)
0927 05:16:01.139621 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.139634 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.139669 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:16:01.139768 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.139775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.139965 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.140181 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:01.141000 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.142942 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.146018 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.147319 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.147483 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.147720 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.147945 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.148098 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":890,"threads_started":1}
I20260927 05:16:01.148428 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.137725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42199 (local address 127.0.0.1:41476)
0927 05:16:01.139304 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.139320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.139421 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.140160 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.140164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.140193 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:01.140774 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.140791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.143117 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.143125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.145829 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.145844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.147541 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.147548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.147723 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.147764 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.147896 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.148026 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"thread_start_us":1124,"threads_started":1}
I20260927 05:16:01.149318 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.135716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.0.1:48386)
0927 05:16:01.138504 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.138537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.138638 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.140418 (+ 1780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.140422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.140453 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:01.141131 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.141141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.142987 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.142993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.146512 (+ 3519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.146526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.148311 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.148317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.148523 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.148555 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.148624 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.148740 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"mutex_wait_us":64,"thread_start_us":843,"threads_started":1}
I20260927 05:16:01.149384 15057 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 (DEFAULT_TABLE table=test-table [id=47787f396db44ac0b6401a62a2969854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:01.150435 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.136361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.97.1:43427)
0927 05:16:01.139887 (+ 3526us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.139895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.139919 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:16:01.140003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.140009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.140202 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.140550 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:01.141355 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.142839 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.146666 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.148060 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.149568 (+ 1508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.149738 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.149944 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.150122 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3369,"thread_start_us":3251,"threads_started":1}
I20260927 05:16:01.150767 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e5aa9b16f7944a4812644e7e63c7211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:01.151938 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 (DEFAULT_TABLE table=test-table [id=47787f396db44ac0b6401a62a2969854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:01.152947 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e5aa9b16f7944a4812644e7e63c7211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:01.153786 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 (DEFAULT_TABLE table=test-table [id=47787f396db44ac0b6401a62a2969854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:01.154953 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e5aa9b16f7944a4812644e7e63c7211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:01.167923 15270 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Bootstrap starting.
I20260927 05:16:01.166762 15269 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Bootstrap starting.
I20260927 05:16:01.169970 15271 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Bootstrap starting.
I20260927 05:16:01.171772 15270 tablet_bootstrap.cc:654] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:01.171981 15269 tablet_bootstrap.cc:654] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:01.172705 15271 tablet_bootstrap.cc:654] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:01.173195 15270 log.cc:824] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:01.173194 15269 log.cc:824] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:01.173734 15271 log.cc:824] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:01.176591 15270 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: No bootstrap required, opened a new log
I20260927 05:16:01.176865 15270 ts_tablet_manager.cc:1397] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 05:16:01.178104 15269 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: No bootstrap required, opened a new log
I20260927 05:16:01.178396 15269 ts_tablet_manager.cc:1397] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 05:16:01.183382 15271 tablet_bootstrap.cc:492] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: No bootstrap required, opened a new log
I20260927 05:16:01.183642 15271 ts_tablet_manager.cc:1397] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260927 05:16:01.187456 15270 raft_consensus.cc:348] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.187873 15270 raft_consensus.cc:374] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:01.187965 15270 raft_consensus.cc:729] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9a203ace134e6983097838089ed521, State: Initialized, Role: FOLLOWER
I20260927 05:16:01.188599 15270 consensus_queue.cc:260] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.197624 15269 raft_consensus.cc:348] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.198056 15269 raft_consensus.cc:374] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:01.198160 15269 raft_consensus.cc:729] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9b5820c924c059627fbab87d50c34, State: Initialized, Role: FOLLOWER
I20260927 05:16:01.202292 15271 raft_consensus.cc:348] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.202713 15271 raft_consensus.cc:374] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:01.202806 15271 raft_consensus.cc:729] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654d35062dff4f8897f9db0ef9681c98, State: Initialized, Role: FOLLOWER
I20260927 05:16:01.203433 15271 consensus_queue.cc:260] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.203567 15269 consensus_queue.cc:260] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.204612 14990 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:16:01.205101 15253 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:16:01.205193 15122 heartbeater.cc:503] Master 127.0.97.62:40991 was elected leader, sending a full tablet report...
I20260927 05:16:01.206281 15269 ts_tablet_manager.cc:1428] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Time spent starting tablet: real 0.028s user 0.009s sys 0.006s
I20260927 05:16:01.207424 15270 ts_tablet_manager.cc:1428] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Time spent starting tablet: real 0.030s user 0.023s sys 0.007s
I20260927 05:16:01.208431 15271 ts_tablet_manager.cc:1428] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Time spent starting tablet: real 0.025s user 0.014s sys 0.000s
W20260927 05:16:01.323690 15254 tablet.cc:2369] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:01.364105 15123 tablet.cc:2369] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:01.437470 14991 tablet.cc:2369] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:16:01.636554 15275 raft_consensus.cc:482] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:16:01.636837 15275 raft_consensus.cc:504] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.638419 15275 leader_election.cc:288] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427), c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199)
I20260927 05:16:01.647583 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.638897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38627 (local address 127.0.97.1:43427)
0927 05:16:01.639115 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.639121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.639881 (+ 760us) server_negotiation.cc:407] Connection header received
0927 05:16:01.640090 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.640095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.640172 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.640273 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:01.641195 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.642307 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.645242 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.646537 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.646691 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.646784 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.647182 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.647311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:16:01.647619 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.638955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33613 (local address 127.0.97.3:42199)
0927 05:16:01.639128 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.639135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.639881 (+ 746us) server_negotiation.cc:407] Connection header received
0927 05:16:01.640089 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.640095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.640172 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.640273 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:01.641187 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.642307 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.645250 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.646552 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.646690 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.646788 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.647182 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.647311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:16:01.647785 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.638782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42199 (local address 127.0.97.2:33613)
0927 05:16:01.639755 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.639770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.639946 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.640308 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.640314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.640347 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:01.640937 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.640956 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.642447 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.642452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.645069 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.645082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.646729 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.646735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.646931 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.647038 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.647135 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.647363 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":405,"spinlock_wait_cycles":30336,"thread_start_us":541,"threads_started":1}
I20260927 05:16:01.648371 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.638723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.97.2:38627)
0927 05:16:01.639739 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:01.639765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:01.639911 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:01.640308 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:01.640314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:01.640352 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:01.641036 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.641051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.642448 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.642452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.645067 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:01.645082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.646780 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.646785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:01.646970 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.647060 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 05:16:01.647135 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 05:16:01.647363 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":285,"spinlock_wait_cycles":4352,"thread_start_us":671,"threads_started":1}
I20260927 05:16:01.648849 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e5aa9b16f7944a4812644e7e63c7211" candidate_uuid: "4f9a203ace134e6983097838089ed521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9b5820c924c059627fbab87d50c34" is_pre_election: true
I20260927 05:16:01.649299 15208 raft_consensus.cc:2454] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f9a203ace134e6983097838089ed521 in term 0.
I20260927 05:16:01.649462 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e5aa9b16f7944a4812644e7e63c7211" candidate_uuid: "4f9a203ace134e6983097838089ed521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654d35062dff4f8897f9db0ef9681c98" is_pre_election: true
I20260927 05:16:01.649921 14945 raft_consensus.cc:2454] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f9a203ace134e6983097838089ed521 in term 0.
I20260927 05:16:01.649992 15010 leader_election.cc:302] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f9a203ace134e6983097838089ed521, c6b9b5820c924c059627fbab87d50c34; no voters:
I20260927 05:16:01.650394 15275 raft_consensus.cc:2781] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:16:01.650586 15275 raft_consensus.cc:482] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:16:01.650713 15275 raft_consensus.cc:3037] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:01.652846 15275 raft_consensus.cc:504] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.653436 15275 leader_election.cc:288] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [CANDIDATE]: Term 1 election: Requested vote from peers 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427), c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199)
I20260927 05:16:01.653807 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e5aa9b16f7944a4812644e7e63c7211" candidate_uuid: "4f9a203ace134e6983097838089ed521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654d35062dff4f8897f9db0ef9681c98"
I20260927 05:16:01.653923 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e5aa9b16f7944a4812644e7e63c7211" candidate_uuid: "4f9a203ace134e6983097838089ed521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9b5820c924c059627fbab87d50c34"
I20260927 05:16:01.653975 14945 raft_consensus.cc:3037] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:01.654145 15208 raft_consensus.cc:3037] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:01.656354 14945 raft_consensus.cc:2454] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9a203ace134e6983097838089ed521 in term 1.
I20260927 05:16:01.656365 15208 raft_consensus.cc:2454] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9a203ace134e6983097838089ed521 in term 1.
I20260927 05:16:01.656747 15013 leader_election.cc:302] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f9a203ace134e6983097838089ed521, 654d35062dff4f8897f9db0ef9681c98; no voters:
I20260927 05:16:01.657027 15275 raft_consensus.cc:2781] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:16:01.657429 15275 raft_consensus.cc:686] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 1 LEADER]: Becoming Leader. State: Replica: 4f9a203ace134e6983097838089ed521, State: Running, Role: LEADER
I20260927 05:16:01.657949 15275 consensus_queue.cc:237] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } }
I20260927 05:16:01.663720 14805 catalog_manager.cc:5495] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f9a203ace134e6983097838089ed521 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4f9a203ace134e6983097838089ed521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9a203ace134e6983097838089ed521" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } health_report { overall_health: UNKNOWN } } }
I20260927 05:16:01.722152 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.716441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.0.97.2:33937)
0927 05:16:01.717212 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.717216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.717232 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:16:01.717310 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.717313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.717381 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.717528 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:16:01.717998 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.718698 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.720193 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.720557 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.720656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.721195 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:16:01.721675 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:16:01.721780 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.721861 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.721916 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":552,"threads_started":1}
I20260927 05:16:01.729753 15208 raft_consensus.cc:1261] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Refusing update from remote peer 4f9a203ace134e6983097838089ed521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:01.729864 14945 raft_consensus.cc:1261] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Refusing update from remote peer 4f9a203ace134e6983097838089ed521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:01.730777 15280 consensus_queue.cc:1035] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b9b5820c924c059627fbab87d50c34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:01.731149 15275 consensus_queue.cc:1035] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654d35062dff4f8897f9db0ef9681c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:01.739621 15288 mvcc.cc:205] Tried to move back new op lower bound from 7333831318435921920 to 7333831318155149312. Current Snapshot: MvccSnapshot[applied={T|T < 7333831318435921920}]
I20260927 05:16:01.756047 14804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:01.756268 14804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:01.759852 14804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Sending DeleteTablet for 3 replicas of tablet 0e5aa9b16f7944a4812644e7e63c7211
I20260927 05:16:01.760847 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58632
I20260927 05:16:01.760885 14925 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48386
I20260927 05:16:01.761168 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e5aa9b16f7944a4812644e7e63c7211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41476
I20260927 05:16:01.761574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14842
I20260927 05:16:01.774343 15302 tablet_replica.cc:330] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: stopping tablet replica
I20260927 05:16:01.775058 15301 tablet_replica.cc:330] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: stopping tablet replica
I20260927 05:16:01.775748 15302 raft_consensus.cc:2229] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:01.776332 15301 raft_consensus.cc:2229] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:16:01.776974 15302 raft_consensus.cc:2258] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:01.777683 15301 raft_consensus.cc:2258] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:01.782439 15301 ts_tablet_manager.cc:1905] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:01.783311 15302 ts_tablet_manager.cc:1905] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:01.783967 15303 tablet_replica.cc:330] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: stopping tablet replica
I20260927 05:16:01.784341 15303 raft_consensus.cc:2229] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:01.784778 15303 raft_consensus.cc:2258] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:01.786484 15303 ts_tablet_manager.cc:1905] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:01.790584 15302 ts_tablet_manager.cc:1918] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:01.790745 15302 log.cc:1196] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/0e5aa9b16f7944a4812644e7e63c7211
I20260927 05:16:01.791287 15302 ts_tablet_manager.cc:1939] T 0e5aa9b16f7944a4812644e7e63c7211 P 654d35062dff4f8897f9db0ef9681c98: Deleting consensus metadata
I20260927 05:16:01.793015 14792 catalog_manager.cc:4841] TS 654d35062dff4f8897f9db0ef9681c98 (127.0.97.1:43427): tablet 0e5aa9b16f7944a4812644e7e63c7211 (table test-table [id=47787f396db44ac0b6401a62a2969854]) successfully deleted
I20260927 05:16:01.797391 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.762386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.97.1:43427)
0927 05:16:01.762567 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.762575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.777822 (+ 15247us) server_negotiation.cc:407] Connection header received
0927 05:16:01.777997 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.778002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.778078 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.778173 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:01.780155 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.780855 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.782811 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.783445 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.783564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.787341 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:01.787354 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:01.787371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:01.787415 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:01.791500 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.794227 (+ 2727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.794239 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.794242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.794399 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.795459 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.795464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.795467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.795928 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:01.796191 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.796795 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.797007 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:16:01.797554 15303 ts_tablet_manager.cc:1918] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:01.797688 15303 log.cc:1196] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/0e5aa9b16f7944a4812644e7e63c7211
I20260927 05:16:01.798280 15303 ts_tablet_manager.cc:1939] T 0e5aa9b16f7944a4812644e7e63c7211 P c6b9b5820c924c059627fbab87d50c34: Deleting consensus metadata
I20260927 05:16:01.799067 14985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:01.799193 15301 ts_tablet_manager.cc:1918] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:01.799341 15301 log.cc:1196] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/0e5aa9b16f7944a4812644e7e63c7211
I20260927 05:16:01.799870 15301 ts_tablet_manager.cc:1939] T 0e5aa9b16f7944a4812644e7e63c7211 P 4f9a203ace134e6983097838089ed521: Deleting consensus metadata
I20260927 05:16:01.800563 14789 catalog_manager.cc:4841] TS c6b9b5820c924c059627fbab87d50c34 (127.0.97.3:42199): tablet 0e5aa9b16f7944a4812644e7e63c7211 (table test-table [id=47787f396db44ac0b6401a62a2969854]) successfully deleted
I20260927 05:16:01.801405 14791 catalog_manager.cc:4841] TS 4f9a203ace134e6983097838089ed521 (127.0.97.2:33937): tablet 0e5aa9b16f7944a4812644e7e63c7211 (table test-table [id=47787f396db44ac0b6401a62a2969854]) successfully deleted
I20260927 05:16:01.891795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14842
I20260927 05:16:01.915951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14994
I20260927 05:16:01.932250 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:01.916741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.0.97.2:33937)
0927 05:16:01.916928 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:01.916934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:01.917127 (+ 193us) server_negotiation.cc:407] Connection header received
0927 05:16:01.917297 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:01.917305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:01.917370 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:01.917454 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:01.919528 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.920343 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.922262 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:01.922685 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:01.922789 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:01.926036 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:01.926049 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:01.926064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:01.926100 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:01.929323 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.930169 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.930194 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.930198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.930355 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:01.930752 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:01.930755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:01.930757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:01.931156 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:01.931405 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:01.931743 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 05:16:01.931917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:16:01.933113 15117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:02.045195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14994
I20260927 05:16:02.064566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15125
I20260927 05:16:02.078724 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:02.065454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.0.97.3:42199)
0927 05:16:02.065666 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:02.065689 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:02.065708 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:16:02.065862 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:02.065868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:02.066011 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:02.066135 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:02.067590 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:02.068334 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:02.069703 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:02.070175 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:02.070331 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:02.072918 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:02.072935 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:02.072949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:02.072989 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:02.076026 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:02.076706 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:02.076713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:02.076716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:02.076850 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:02.077229 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:02.077235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:02.077239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:02.077564 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:02.077857 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:02.078153 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 05:16:02.078362 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:16:02.079610 15248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:02.167593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15125
I20260927 05:16:02.184022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 14772
I20260927 05:16:02.185288 14835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:02.296216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 14772
I20260927 05:16:02.317272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--webserver_interface=127.0.97.62
--webserver_port=44291
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:02.633136 15309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:02.633394 15309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:02.633572 15309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:02.640537 15309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:16:02.640631 15309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:02.640726 15309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:02.640828 15309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:16:02.640897 15309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:16:02.650262 15309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:02.651489 15309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:02.652722 15309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:02.664206 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:02.664147 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:02.664242 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:02.665114 15309 server_base.cc:1034] running on GCE node
I20260927 05:16:02.665930 15309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:02.667371 15309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:02.668566 15309 hybrid_clock.cc:648] HybridClock initialized: now 1790486162668506 us; error 46 us; skew 500 ppm
I20260927 05:16:02.668915 15309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:02 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:02.671950 15309 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:16:02.673323 15309 webserver.cc:460] Webserver started at http://127.0.97.62:44291/ using document root <none> and password file <none>
I20260927 05:16:02.673985 15309 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:02.674129 15309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:02.679394 15309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:02.682118 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:02.683625 15309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:02.683812 15309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "012b4dc648a14bb9a38600ecd433dcde"
format_stamp: "Formatted at 2026-09-27 05:15:56 on dist-test-slave-1t6p"
server_key: "388fa87410a643f883bba383210b31e2"
server_key_iv: "4ae817edaf3488df03bd701330ee13d4"
server_key_version: "encryptionkey@0"
I20260927 05:16:02.684768 15309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:02.706820 15309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:02.710719 15309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:02.711051 15309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:02.729723 15309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40991
I20260927 05:16:02.729773 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40991 every 8 connection(s)
I20260927 05:16:02.731364 15309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:16:02.737151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15309
I20260927 05:16:02.738193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43427
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=39185
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:16:02.738492 15376 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:16:02.743712 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap starting.
I20260927 05:16:02.764542 15376 log.cc:824] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:02.780838 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:16:02.781401 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde: Bootstrap complete.
I20260927 05:16:02.791162 15376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:16:02.792014 15376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Initialized, Role: FOLLOWER
I20260927 05:16:02.792712 15376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:16:02.792949 15376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:16:02.793066 15376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:16:02.793256 15376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:16:02.797062 15376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:16:02.797571 15376 leader_election.cc:302] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012b4dc648a14bb9a38600ecd433dcde; no voters:
I20260927 05:16:02.798249 15376 leader_election.cc:288] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:16:02.798383 15380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:16:02.799614 15380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [term 3 LEADER]: Becoming Leader. State: Replica: 012b4dc648a14bb9a38600ecd433dcde, State: Running, Role: LEADER
I20260927 05:16:02.800151 15380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } }
I20260927 05:16:02.800478 15376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:16:02.802733 15382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012b4dc648a14bb9a38600ecd433dcde. Latest consensus state: current_term: 3 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:16:02.803197 15382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:02.803138 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "012b4dc648a14bb9a38600ecd433dcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012b4dc648a14bb9a38600ecd433dcde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40991 } } }
I20260927 05:16:02.803534 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012b4dc648a14bb9a38600ecd433dcde [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:02.815299 15388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:16:02.826188 15388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:16:02.827673 15388 catalog_manager.cc:1259] Loaded cluster ID: 0de358c8977146628782bb2d31ccb74c
I20260927 05:16:02.827777 15388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 05:16:02.829696 15388 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4b78cd79ad google::LogMessage::Fail() at ??:0
@ 0x7f4b78cda88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4b78cd74de google::LogMessage::Flush() at ??:0
@ 0x7f4b78cd8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4b8e70a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4b8e707976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4b8e78f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4b8e78f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4b7b923345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4b7b92691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4b7b926799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4b7b901eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4b7ebdb6db start_thread at ??:0
@ 0x7f4b77b7161f clone at ??:0
W20260927 05:16:03.129431 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:03.129698 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:03.129861 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:03.136973 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:03.137096 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:03.137269 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:16:03.146708 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39185
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:03.147954 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:03.149212 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:03.165761 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:03.165792 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:03.165778 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:03.167136 15378 server_base.cc:1034] running on GCE node
I20260927 05:16:03.167721 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:03.168629 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:03.169873 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790486163169835 us; error 45 us; skew 500 ppm
I20260927 05:16:03.170209 15378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:03 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:03.173297 15378 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:16:03.174731 15378 webserver.cc:460] Webserver started at http://127.0.97.1:39185/ using document root <none> and password file <none>
I20260927 05:16:03.175462 15378 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:03.175616 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:03.180686 15378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:16:03.183338 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.184651 15378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:03.184839 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "654d35062dff4f8897f9db0ef9681c98"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "48f124b58389d784f47fca8cba62cda2"
server_key_iv: "d90a67b0bc1bda0c5605c4f7df48701d"
server_key_version: "encryptionkey@0"
I20260927 05:16:03.185810 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:03.199752 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:03.203970 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:03.204258 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:03.205216 15378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:03.206511 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:03.206645 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.206908 15378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:03.207084 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.240808 15378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43427
I20260927 05:16:03.240895 15539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43427 every 8 connection(s)
I20260927 05:16:03.242231 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:16:03.243458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15378
I20260927 05:16:03.244211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33937
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=43553
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:16:03.247684 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.245739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:33299)
0927 05:16:03.246513 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.246700 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":328,"threads_started":1}
W20260927 05:16:03.248464 15540 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40991 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40991: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
I20260927 05:16:03.249305 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.248971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:42905)
0927 05:16:03.249133 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.249206 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:16:03.250205 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.249880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:51251)
0927 05:16:03.250024 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.250105 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:16:03.250455 15540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:16:03.565371 15543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:03.565652 15543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:03.565887 15543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:03.572877 15543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:03.572968 15543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:03.573196 15543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:16:03.582351 15543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43553
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:03.583513 15543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:03.584793 15543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:03.597386 15548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:03.597410 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:03.597464 15549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:03.598474 15543 server_base.cc:1034] running on GCE node
I20260927 05:16:03.599071 15543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:03.599939 15543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:03.601187 15543 hybrid_clock.cc:648] HybridClock initialized: now 1790486163601136 us; error 36 us; skew 500 ppm
I20260927 05:16:03.601496 15543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:03 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:03.604669 15543 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:16:03.606115 15543 webserver.cc:460] Webserver started at http://127.0.97.2:43553/ using document root <none> and password file <none>
I20260927 05:16:03.606760 15543 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:03.606913 15543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:03.611975 15543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:16:03.614473 15558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.615804 15543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:03.615972 15543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "4f9a203ace134e6983097838089ed521"
format_stamp: "Formatted at 2026-09-27 05:15:57 on dist-test-slave-1t6p"
server_key: "95b8cc677c11e930b0dcd77381b29654"
server_key_iv: "d4f4a98af36d4ed7df3a4c8198534552"
server_key_version: "encryptionkey@0"
I20260927 05:16:03.616902 15543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:03.647228 15543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:03.651535 15543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:03.651810 15543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:03.652733 15543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:03.654008 15543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:03.654141 15543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.654276 15543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:03.654371 15543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:03.690968 15543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33937
I20260927 05:16:03.691085 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33937 every 8 connection(s)
I20260927 05:16:03.692389 15543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:16:03.696833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15543
I20260927 05:16:03.696995 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.695299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:60437)
0927 05:16:03.695983 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.696170 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":282,"threads_started":1}
W20260927 05:16:03.697575 15671 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40991 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40991: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
I20260927 05:16:03.697590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42199
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40991
--webserver_port=45897
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:16:03.698441 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.698125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:34745)
0927 05:16:03.698267 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.698337 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:16:03.699328 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:03.699023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:39235)
0927 05:16:03.699155 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:03.699223 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:16:03.699626 15671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:16:04.015733 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:04.016003 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:04.016155 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:04.023552 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:04.023659 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:04.023865 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:16:04.033355 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45897
--tserver_master_addrs=127.0.97.62:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:04.034605 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:04.035925 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:04.049016 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:04.049010 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:04.049041 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:04.049885 15674 server_base.cc:1034] running on GCE node
I20260927 05:16:04.050501 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:04.051502 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:04.052727 15674 hybrid_clock.cc:648] HybridClock initialized: now 1790486164052675 us; error 55 us; skew 500 ppm
I20260927 05:16:04.053021 15674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:04 dist-test-slave-1t6p krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:04.056012 15674 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:16:04.057402 15674 webserver.cc:460] Webserver started at http://127.0.97.3:45897/ using document root <none> and password file <none>
I20260927 05:16:04.058049 15674 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:04.058208 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:04.063261 15674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:16:04.065801 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:04.067253 15674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:04.067380 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "c6b9b5820c924c059627fbab87d50c34"
format_stamp: "Formatted at 2026-09-27 05:15:58 on dist-test-slave-1t6p"
server_key: "91eda73b8546cbfe673ad320727ed56f"
server_key_iv: "2398e11ac30f5227065399a617ef6273"
server_key_version: "encryptionkey@0"
I20260927 05:16:04.068250 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:04.100611 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:04.104401 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:04.104631 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:04.105616 15674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:04.106972 15674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:04.107090 15674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:04.107234 15674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:04.107331 15674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:04.140367 15674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42199
I20260927 05:16:04.140453 15801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42199 every 8 connection(s)
I20260927 05:16:04.141743 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:16:04.146260 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:04.144549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:40749)
0927 05:16:04.145237 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:04.145427 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":285,"threads_started":1}
W20260927 05:16:04.146888 15802 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40991 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40991: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
I20260927 05:16:04.147760 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:04.147397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:39165)
0927 05:16:04.147545 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:04.147651 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:16:04.148672 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:04.148279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:54721)
0927 05:16:04.148436 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:04.148526 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:16:04.148933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15674
W20260927 05:16:04.149161 15802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:16:04.251868 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:04.251109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:33961)
0927 05:16:04.251557 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:04.251653 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260927 05:16:04.700897 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:04.700126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:42315)
0927 05:16:04.700574 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:04.700673 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260927 05:16:05.150363 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:05.149652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:54325)
0927 05:16:05.150048 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:05.150143 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":195,"threads_started":1}
I20260927 05:16:05.253485 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:05.252831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:45103)
0927 05:16:05.253250 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:05.253331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":209,"threads_started":1}
I20260927 05:16:05.702430 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:05.701755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:46327)
0927 05:16:05.702196 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:05.702273 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260927 05:16:06.151991 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:06.151259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:54591)
0927 05:16:06.151773 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:06.151842 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260927 05:16:06.254859 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:06.254223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:54887)
0927 05:16:06.254629 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:06.254704 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
I20260927 05:16:06.703994 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:06.703362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:49971)
0927 05:16:06.703765 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:06.703837 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":222,"threads_started":1}
I20260927 05:16:07.153417 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:07.152768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:36563)
0927 05:16:07.153191 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:07.153262 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":237,"threads_started":1}
I20260927 05:16:07.256367 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:07.255684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:38401)
0927 05:16:07.256116 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:07.256190 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260927 05:16:07.705440 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:07.704762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:38325)
0927 05:16:07.705206 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:07.705279 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260927 05:16:08.154949 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:08.154295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:60091)
0927 05:16:08.154710 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:08.154781 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":222,"threads_started":1}
I20260927 05:16:08.258102 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:08.257336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:56981)
0927 05:16:08.257857 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:08.257932 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260927 05:16:08.707259 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:08.706486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:45781)
0927 05:16:08.706924 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:08.707070 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260927 05:16:09.156831 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:09.155997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:41447)
0927 05:16:09.156549 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:09.156642 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":287,"threads_started":1}
I20260927 05:16:09.259699 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:09.258953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:50883)
0927 05:16:09.259449 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:09.259537 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":293,"threads_started":1}
I20260927 05:16:09.708827 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:09.708133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:37657)
0927 05:16:09.708576 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:09.708661 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260927 05:16:10.158319 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:10.157657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:53377)
0927 05:16:10.158082 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:10.158167 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260927 05:16:10.261339 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:10.260532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:50461)
0927 05:16:10.261031 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:10.261113 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":231,"threads_started":1}
I20260927 05:16:10.710479 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:10.709743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:39283)
0927 05:16:10.710197 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:10.710282 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":199,"threads_started":1}
I20260927 05:16:11.159967 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:11.159212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:54701)
0927 05:16:11.159664 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:11.159761 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":189,"threads_started":1}
I20260927 05:16:11.262835 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:11.262190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:57219)
0927 05:16:11.262608 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:11.262681 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260927 05:16:11.712049 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:11.711405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:36861)
0927 05:16:11.711826 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:11.711900 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260927 05:16:12.161644 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:12.160853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:45311)
0927 05:16:12.161300 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:12.161380 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260927 05:16:12.264303 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:12.263613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:56655)
0927 05:16:12.264062 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:12.264137 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":220,"threads_started":1}
I20260927 05:16:12.713536 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:12.712868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:47667)
0927 05:16:12.713306 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:12.713377 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260927 05:16:13.163357 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:13.162461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:49689)
0927 05:16:13.162991 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:13.163173 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260927 05:16:13.265815 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:13.265147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:42929)
0927 05:16:13.265585 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:13.265659 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260927 05:16:13.715198 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:13.714426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:59389)
0927 05:16:13.714886 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:13.714964 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260927 05:16:14.165071 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:14.164245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:57213)
0927 05:16:14.164768 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:14.164845 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":320,"threads_started":1}
I20260927 05:16:14.267416 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:14.266700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:46005)
0927 05:16:14.267163 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:14.267248 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":204,"threads_started":1}
I20260927 05:16:14.716790 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:14.716082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:41295)
0927 05:16:14.716534 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:14.716621 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":222,"threads_started":1}
I20260927 05:16:15.166697 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:15.165945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:48923)
0927 05:16:15.166376 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:15.166458 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260927 05:16:15.268853 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:15.268194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:48203)
0927 05:16:15.268633 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:15.268707 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
I20260927 05:16:15.718327 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:15.717668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:38755)
0927 05:16:15.718094 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:15.718169 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":224,"threads_started":1}
I20260927 05:16:16.168290 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:16.167652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:58235)
0927 05:16:16.168057 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:16.168137 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":212,"threads_started":1}
I20260927 05:16:16.270454 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:16.269705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:60537)
0927 05:16:16.270164 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:16.270249 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":224,"threads_started":1}
I20260927 05:16:16.719993 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:16.719234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:51767)
0927 05:16:16.719641 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:16.719736 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260927 05:16:17.170042 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:17.169153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:45169)
0927 05:16:17.169706 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:17.169792 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
I20260927 05:16:17.272249 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:17.271447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:32997)
0927 05:16:17.271981 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:17.272071 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":196,"threads_started":1}
I20260927 05:16:17.721503 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:17.720877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:57947)
0927 05:16:17.721263 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:17.721333 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":192,"threads_started":1}
I20260927 05:16:18.171803 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:18.170985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:34189)
0927 05:16:18.171526 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:18.171616 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260927 05:16:18.273917 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:18.273241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.1:56599)
0927 05:16:18.273653 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:18.273742 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":214,"threads_started":1}
I20260927 05:16:18.723230 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:18.722487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.2:48275)
0927 05:16:18.722884 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:18.722966 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":209,"threads_started":1}
I20260927 05:16:19.156966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15378
I20260927 05:16:19.179324 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:19.173112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40991 (local address 127.0.97.3:53249)
0927 05:16:19.178661 (+ 5549us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:19.178764 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5421,"thread_start_us":5319,"threads_started":1}
I20260927 05:16:19.180380 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:19.158076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.97.1:43427)
0927 05:16:19.158708 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:19.158728 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:19.158777 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:16:19.160891 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:19.160901 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:19.161336 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:19.161645 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:19.163068 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.164297 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.165591 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.166237 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.166367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:19.170099 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:19.170112 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:19.170136 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:19.170193 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:19.174982 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.175940 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.175949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.175959 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.176164 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.176702 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.176709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.176715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.177490 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:19.178956 (+ 1466us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:19.179328 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:16:19.179849 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":251,"threads_started":1}
I20260927 05:16:19.182539 15535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:19.280009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15378
I20260927 05:16:19.297012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15543
I20260927 05:16:19.314426 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:19.297875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.0.97.2:33937)
0927 05:16:19.298400 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:19.298415 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:19.298448 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:16:19.300010 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:19.300020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:19.300436 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:19.300804 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:19.302170 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.303490 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.304749 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.305420 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.305631 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:19.308297 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:19.308304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:19.308318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:19.308356 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:19.311305 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.311949 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.311957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.311966 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.312135 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.312515 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.312521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.312524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.313003 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:19.313298 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:19.313752 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 05:16:19.314150 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260927 05:16:19.315763 15666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:19.398972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15543
I20260927 05:16:19.416955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15674
I20260927 05:16:19.434269 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:19.417825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.97.3:42199)
0927 05:16:19.418371 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:19.418384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:19.418423 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:16:19.420101 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:19.420110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:19.420531 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:19.420896 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:19.422315 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.423692 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.424905 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:19.425579 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:19.425769 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:19.428306 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:19.428313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:19.428327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:19.428361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:19.431191 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.431803 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.431811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.431826 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.431984 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:19.432359 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:19.432365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:19.432368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:19.432852 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:19.433142 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:19.433593 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 05:16:19.433977 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":253,"threads_started":1}
I20260927 05:16:19.435631 15797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:19.517148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15674
2026-09-27T05:16:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24675 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:16:19 dist-test-slave-1t6p krb5kdc[15860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:16:19 dist-test-slave-1t6p krb5kdc[15860](info): set up 2 sockets
Sep 27 05:16:19 dist-test-slave-1t6p krb5kdc[15860](info): commencing operation
krb5kdc: starting...
W20260927 05:16:20.607600 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:16:20 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:16:20.830171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:21.144157 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:21.144416 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:21.144567 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:21.151477 15873 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:16:21.151567 15873 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:16:21.151644 15873 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:16:21.151741 15873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:16:21.151814 15873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:21.151917 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:21.151983 15873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:16:21.152064 15873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:16:21.161731 15873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:21.162992 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:21.164359 15873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:21.176247 15878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:21.176378 15873 server_base.cc:1034] running on GCE node
W20260927 05:16:21.176342 15879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:21.176299 15881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:21.177495 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:21.178903 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:21.180145 15873 hybrid_clock.cc:648] HybridClock initialized: now 1790486181180086 us; error 35 us; skew 500 ppm
I20260927 05:16:21.180475 15873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:21 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:21.186441 15873 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:16:21.187899 15873 webserver.cc:460] Webserver started at http://127.0.97.62:38489/ using document root <none> and password file <none>
I20260927 05:16:21.188649 15873 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:21.188779 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:21.189083 15873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:16:21.191996 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "fc71c246477f4b14a9c822cae7167adc"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "3a8aa291d71f057a96e2b79dcb0226fa"
server_key_iv: "361e775a6621ff602cb313de3316d585"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.192767 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc71c246477f4b14a9c822cae7167adc"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "3a8aa291d71f057a96e2b79dcb0226fa"
server_key_iv: "361e775a6621ff602cb313de3316d585"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.197762 15873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:16:21.201139 15888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:21.202394 15873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 05:16:21.202566 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "fc71c246477f4b14a9c822cae7167adc"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "3a8aa291d71f057a96e2b79dcb0226fa"
server_key_iv: "361e775a6621ff602cb313de3316d585"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.202797 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:21.234151 15873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:21.237922 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:21.238135 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:21.255609 15873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40863
I20260927 05:16:21.255797 15939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40863 every 8 connection(s)
I20260927 05:16:21.257002 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:16:21.259954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15873
I20260927 05:16:21.260162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:16:21.260826 388 external_mini_cluster.cc:1428] Setting key 10a088bbfd352f50bcc89db7e1280cd0
I20260927 05:16:21.260825 15940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:21.274578 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap starting.
Sep 27 05:16:21 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:21.277990 15940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:21.279364 15940 log.cc:824] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:21.281997 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: No bootstrap required, opened a new log
I20260927 05:16:21.284009 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:21.262736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.0.97.62:40863)
0927 05:16:21.263577 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:21.263592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:21.263645 (+ 53us) server_negotiation.cc:407] Connection header received
0927 05:16:21.265683 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:21.265711 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:21.266196 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:21.266615 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:21.267901 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.269840 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.271239 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.272316 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.272508 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:21.276394 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:21.276404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:21.276420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:21.276462 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:21.280260 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:21.280918 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:21.280932 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:21.280938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:21.281096 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:21.281483 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:21.281491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:21.281493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:21.281961 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:21.282279 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:21.282673 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 05:16:21.283108 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":367,"threads_started":1}
I20260927 05:16:21.289824 15940 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:21.290211 15940 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:21.290319 15940 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Initialized, Role: FOLLOWER
I20260927 05:16:21.290832 15940 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:21.291097 15940 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:16:21.291241 15940 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:16:21.291410 15940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:21.293764 15940 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:21.294212 15940 leader_election.cc:302] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc71c246477f4b14a9c822cae7167adc; no voters:
I20260927 05:16:21.294842 15940 leader_election.cc:288] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:16:21.294929 15945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:16:21.295635 15945 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 LEADER]: Becoming Leader. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Running, Role: LEADER
I20260927 05:16:21.296149 15945 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:21.296773 15940 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:16:21.299398 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:16:21.299921 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:21.299837 15947 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc71c246477f4b14a9c822cae7167adc. Latest consensus state: current_term: 1 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:16:21.300149 15947 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:21.305488 15952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:16:21.310614 15952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:16:21.318765 15952 catalog_manager.cc:1347] Generated new cluster ID: c093d34e7894455c8a73e0764c34e5b7
I20260927 05:16:21.318886 15952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:16:21.336212 15952 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:16:21.337944 15952 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:16:21.351722 15952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Generated new TSK 0
I20260927 05:16:21.352698 15952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:16:21.410591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:21.734540 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:21.734808 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:21.734896 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:21.742161 15966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:21.742252 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:21.742355 15966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:16:21.751745 15966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:21.752837 15966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:21.754027 15966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:21.766928 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:21.767031 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:21.766963 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:21.767647 15966 server_base.cc:1034] running on GCE node
I20260927 05:16:21.768174 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:21.769047 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:21.770247 15966 hybrid_clock.cc:648] HybridClock initialized: now 1790486181770190 us; error 45 us; skew 500 ppm
I20260927 05:16:21.770502 15966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:21 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:21.773643 15966 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:16:21.775138 15966 webserver.cc:460] Webserver started at http://127.0.97.1:40481/ using document root <none> and password file <none>
I20260927 05:16:21.775781 15966 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:21.775923 15966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:21.776232 15966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:16:21.779346 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "adf71d7882894d47aa381c553482e363"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "5463f538e851b603fbb3e184d908c243"
server_key_iv: "eec71e4df392eab81e7cdfa5d509fa52"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.780151 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "adf71d7882894d47aa381c553482e363"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "5463f538e851b603fbb3e184d908c243"
server_key_iv: "eec71e4df392eab81e7cdfa5d509fa52"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.785143 15966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:16:21.788453 15981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:21.789800 15966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:16:21.789971 15966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "adf71d7882894d47aa381c553482e363"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "5463f538e851b603fbb3e184d908c243"
server_key_iv: "eec71e4df392eab81e7cdfa5d509fa52"
server_key_version: "encryptionkey@0"
I20260927 05:16:21.790189 15966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:21.804147 15966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:21.808496 15966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:21.808804 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:21.809717 15966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:21.810945 15966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:21.811067 15966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:21.811210 15966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:21.811303 15966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:21.842361 15966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43335
I20260927 05:16:21.842403 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43335 every 8 connection(s)
I20260927 05:16:21.843590 15966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:16:21.850872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 15966
I20260927 05:16:21.851136 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:16:21.851776 388 external_mini_cluster.cc:1428] Setting key 7e49df12c27b9c29d199cbaef322e869
Sep 27 05:16:21 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:21.863935 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:21.846117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33589 (local address 127.0.97.62:40863)
0927 05:16:21.846585 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:21.846591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:21.847528 (+ 937us) server_negotiation.cc:407] Connection header received
0927 05:16:21.849257 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:21.849261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:21.849327 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:21.849424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:21.851000 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.852067 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.853939 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.854364 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.854459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:21.858379 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:21.858384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:21.858387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:21.858417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:21.860986 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:21.861656 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:21.861662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:21.861667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:21.861820 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:21.862283 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:21.862286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:21.862288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:21.862539 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:21.862675 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:21.863517 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 05:16:21.863694 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260927 05:16:21.865353 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:21.846584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:33589)
0927 05:16:21.847377 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:21.847419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:21.849018 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:21.849625 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:21.849632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:21.850098 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:21.850796 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:21.850820 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.852216 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.852221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:21.853802 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:21.853816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:21.854580 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:21.854589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:21.854916 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:21.855634 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:21.855668 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:21.858177 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:21.861135 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:21.861141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:21.861154 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:21.861534 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:21.861937 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:21.861940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:21.861943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:21.862168 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:21.862663 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:21.862674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:21.863138 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 05:16:21.863425 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 05:16:21.863967 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":353,"threads_started":1}
I20260927 05:16:21.866899 16094 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:21.867223 16094 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:21.867928 16094 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:21.869896 15905 ts_manager.cc:194] Registered new tserver with Master: adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:21.871347 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33589
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:16:21.903451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:22.217355 16100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:22.217618 16100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:22.217710 16100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:22.224608 16100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:22.224704 16100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:22.224809 16100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:16:22.234066 16100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:22.235244 16100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:22.236441 16100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:22.248752 16105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:22.248849 16108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:22.248871 16106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:22.249457 16100 server_base.cc:1034] running on GCE node
I20260927 05:16:22.250016 16100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:22.250852 16100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:22.252094 16100 hybrid_clock.cc:648] HybridClock initialized: now 1790486182252052 us; error 46 us; skew 500 ppm
I20260927 05:16:22.252424 16100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:22 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:22.255506 16100 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:16:22.260056 16100 webserver.cc:460] Webserver started at http://127.0.97.2:39205/ using document root <none> and password file <none>
I20260927 05:16:22.260690 16100 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:22.260830 16100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:22.261138 16100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:16:22.264153 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3626704972a4f33a6797e0a7ea89b8e"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "257d8798a639da52f459282e4a48e224"
server_key_iv: "f3f9fde00e1d8ab9ab0b7dddfdcda358"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.264881 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3626704972a4f33a6797e0a7ea89b8e"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "257d8798a639da52f459282e4a48e224"
server_key_iv: "f3f9fde00e1d8ab9ab0b7dddfdcda358"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.269760 16100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:16:22.273018 16115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.274313 16100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:16:22.274472 16100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f3626704972a4f33a6797e0a7ea89b8e"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "257d8798a639da52f459282e4a48e224"
server_key_iv: "f3f9fde00e1d8ab9ab0b7dddfdcda358"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.274631 16100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:22.305995 16100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:22.310283 16100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:22.310567 16100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:22.311530 16100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:22.312875 16100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:22.312947 16100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.313026 16100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:22.313077 16100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.345112 16100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45199
I20260927 05:16:22.345199 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45199 every 8 connection(s)
I20260927 05:16:22.346385 16100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:16:22.355161 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16100
I20260927 05:16:22.355356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:16:22.355794 388 external_mini_cluster.cc:1428] Setting key 0f57adb28c13f078de7302046062c80e
Sep 27 05:16:22 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:22.366322 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.348849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56147 (local address 127.0.97.62:40863)
0927 05:16:22.349069 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.349076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.350293 (+ 1217us) server_negotiation.cc:407] Connection header received
0927 05:16:22.352135 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.352139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.352207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.352293 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:22.353690 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.354366 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.356082 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.356598 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.356748 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.360493 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:22.360498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:22.360501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:22.360533 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:22.363354 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.364065 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.364069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.364072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.364196 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.364633 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.364642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.364645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.364906 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:22.365069 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.365738 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.365960 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:16:22.367313 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.349383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:56147)
0927 05:16:22.350132 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:22.350178 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:22.351894 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:22.352487 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:22.352495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:22.352991 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:22.353552 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.353569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.354497 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.354500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.355945 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.355957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.356829 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.356836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.357061 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.357747 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:22.357776 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:22.360280 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:22.363499 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:22.363508 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:22.363523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:22.363925 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:22.364310 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:22.364315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:22.364319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:22.364506 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:22.365035 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:22.365043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:22.365361 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 05:16:22.365627 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 05:16:22.366160 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":349,"threads_started":1}
I20260927 05:16:22.368717 16228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:22.369019 16228 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:22.369829 16228 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:22.371420 15905 ts_manager.cc:194] Registered new tserver with Master: f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2:45199)
I20260927 05:16:22.372742 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56147
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:16:22.408244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:22.726092 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:22.726374 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:22.726527 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:22.733531 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:22.733625 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:22.733861 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:16:22.743868 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:22.745041 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:22.746294 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:22.759402 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:22.759356 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:22.759445 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:22.760571 16233 server_base.cc:1034] running on GCE node
I20260927 05:16:22.761183 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:22.762094 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:22.763314 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790486182763279 us; error 58 us; skew 500 ppm
I20260927 05:16:22.763623 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:22 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:22.766669 16233 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:16:22.768105 16233 webserver.cc:460] Webserver started at http://127.0.97.3:45279/ using document root <none> and password file <none>
I20260927 05:16:22.768694 16233 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:22.768795 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:22.769029 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:16:22.771796 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "ac0ce6af33491cc7b2e4847ca1cfe2e1"
server_key_iv: "3f250499a6c73df58f496a0b6cc2d79f"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.772468 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "ac0ce6af33491cc7b2e4847ca1cfe2e1"
server_key_iv: "3f250499a6c73df58f496a0b6cc2d79f"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.777446 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:16:22.780809 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.782083 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:16:22.782224 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "ac0ce6af33491cc7b2e4847ca1cfe2e1"
server_key_iv: "3f250499a6c73df58f496a0b6cc2d79f"
server_key_version: "encryptionkey@0"
I20260927 05:16:22.782385 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:22.818419 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:22.822553 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:22.822791 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:22.823654 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:22.824900 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:22.824970 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.825055 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:22.825099 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:22.857409 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43593
I20260927 05:16:22.857493 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43593 every 8 connection(s)
I20260927 05:16:22.858700 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:16:22.868712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16233
I20260927 05:16:22.868904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:16:22.869333 388 external_mini_cluster.cc:1428] Setting key 8626cc85196336ed98ceae568be5c8cb
Sep 27 05:16:22 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:22.874018 16094 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:22.877765 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.861178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53009 (local address 127.0.97.62:40863)
0927 05:16:22.861386 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.861392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.862524 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 05:16:22.864287 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.864291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.864355 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.864440 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:22.865805 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.866524 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.867992 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.868703 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.868838 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.872663 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:22.872669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:22.872684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:22.872719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:22.875260 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.875971 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.875975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.875977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.876094 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.876447 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.876450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.876452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.876718 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:22.876820 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.877377 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.877517 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:16:22.878664 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.861717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:53009)
0927 05:16:22.862370 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:22.862404 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:22.864048 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:22.864632 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:22.864639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:22.865135 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:22.865659 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.865675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.866636 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.866639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.867877 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.867889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.868941 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.868963 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.869172 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.869886 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:22.869920 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:22.872482 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:22.875416 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:22.875424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:22.875437 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:22.875860 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:22.876185 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:22.876187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:22.876190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:22.876354 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:22.876815 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:22.876820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:22.877079 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 05:16:22.877282 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 05:16:22.877652 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
I20260927 05:16:22.879966 16361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:22.880245 16361 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:22.880894 16361 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:22.882138 15905 ts_manager.cc:194] Registered new tserver with Master: cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593)
I20260927 05:16:22.883149 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53009
I20260927 05:16:22.884527 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:16:22.901201 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.889160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.0.97.62:40863)
0927 05:16:22.889360 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.889369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.889646 (+ 277us) server_negotiation.cc:407] Connection header received
0927 05:16:22.889784 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.889787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.889840 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.889914 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:22.891329 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.892003 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.893337 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.893736 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.893854 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.896270 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:22.896274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:22.896277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:22.896305 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:22.898734 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.899487 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.899491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.899493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.899605 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:22.899952 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:22.899955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:22.899957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:22.900254 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:22.900432 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.900697 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.900867 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:16:22.904858 15905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:16:22.911877 15905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:16:22.943056 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.923244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.97.3:43593)
0927 05:16:22.935353 (+ 12109us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.935362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.935398 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:16:22.935498 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.935509 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.935787 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.936712 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:22.936991 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.938353 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.940685 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.941772 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.941918 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.942159 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.942535 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.942693 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11909,"thread_start_us":11761,"threads_started":1}
I20260927 05:16:22.943086 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.922855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43593 (local address 127.0.0.1:34788)
0927 05:16:22.923644 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:22.923689 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:22.923851 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:22.936031 (+ 12180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:22.936038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:22.936071 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:22.936514 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.936526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.938479 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.938487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.940528 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.940539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.941983 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.941993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.942287 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.942341 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 05:16:22.942476 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 05:16:22.942651 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":3,"thread_start_us":338,"threads_started":1}
I20260927 05:16:22.945004 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.924691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45199 (local address 127.0.0.1:42328)
0927 05:16:22.935540 (+ 10849us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:22.935556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:22.935663 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:22.937144 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:22.937149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:22.937178 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:22.937697 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.937717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.939559 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.939563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.942151 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.942160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.944161 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.944167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.944336 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.944366 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:16:22.944473 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:16:22.944588 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10728,"mutex_wait_us":121,"thread_start_us":9880,"threads_started":1}
I20260927 05:16:22.945128 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.923853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.0.97.1:43335)
0927 05:16:22.930771 (+ 6918us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.930796 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.935078 (+ 4282us) server_negotiation.cc:407] Connection header received
0927 05:16:22.936186 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.936194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.936421 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.936578 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:22.937727 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.939384 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.942305 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.944017 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.944130 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.944373 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.944658 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.944811 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":358,"threads_started":1}
I20260927 05:16:22.945156 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.923065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43335 (local address 127.0.0.1:55522)
0927 05:16:22.934933 (+ 11868us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:22.934956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:22.935813 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:22.937046 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:22.937053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:22.937083 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:22.937532 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.937542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.939557 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.939562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.942139 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:22.942153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.944283 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.944289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:22.944480 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.944502 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:16:22.944616 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 05:16:22.944743 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11693,"mutex_wait_us":113,"spinlock_wait_cycles":6528,"thread_start_us":12746,"threads_started":1}
I20260927 05:16:22.945781 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:22.924963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.0.97.2:45199)
0927 05:16:22.936585 (+ 11622us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:22.936593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:22.936637 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:16:22.936743 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:22.936750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:22.936948 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:22.937138 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:22.937873 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.939408 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.942334 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:22.943692 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:22.943835 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:22.944025 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:22.945280 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0927 05:16:22.945437 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11455,"thread_start_us":11803,"threads_started":1}
I20260927 05:16:22.947577 16296 tablet_service.cc:1467] Processing CreateTablet for tablet 7d03c46899cd46dc9653a443a872e75e (DEFAULT_TABLE table=test-table [id=b06e24bb6459496e9c535c5aae043743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:22.948647 16029 tablet_service.cc:1467] Processing CreateTablet for tablet 7d03c46899cd46dc9653a443a872e75e (DEFAULT_TABLE table=test-table [id=b06e24bb6459496e9c535c5aae043743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:22.949118 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d03c46899cd46dc9653a443a872e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:22.949117 16163 tablet_service.cc:1467] Processing CreateTablet for tablet 7d03c46899cd46dc9653a443a872e75e (DEFAULT_TABLE table=test-table [id=b06e24bb6459496e9c535c5aae043743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:22.949838 16029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d03c46899cd46dc9653a443a872e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:22.950456 16163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d03c46899cd46dc9653a443a872e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:22.961216 16379 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Bootstrap starting.
I20260927 05:16:22.963521 16380 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Bootstrap starting.
I20260927 05:16:22.965065 16379 tablet_bootstrap.cc:654] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:22.965451 16381 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Bootstrap starting.
I20260927 05:16:22.966836 16379 log.cc:824] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:22.967512 16380 tablet_bootstrap.cc:654] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:22.968531 16381 tablet_bootstrap.cc:654] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:22.968552 16380 log.cc:824] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:22.969660 16381 log.cc:824] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:22.969877 16379 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: No bootstrap required, opened a new log
I20260927 05:16:22.970144 16379 ts_tablet_manager.cc:1397] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 05:16:22.971484 16380 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: No bootstrap required, opened a new log
I20260927 05:16:22.971738 16380 ts_tablet_manager.cc:1397] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 05:16:22.972254 16381 tablet_bootstrap.cc:492] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: No bootstrap required, opened a new log
I20260927 05:16:22.972501 16381 ts_tablet_manager.cc:1397] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 05:16:22.979852 16381 raft_consensus.cc:348] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.980206 16381 raft_consensus.cc:374] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:22.980325 16381 raft_consensus.cc:729] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3626704972a4f33a6797e0a7ea89b8e, State: Initialized, Role: FOLLOWER
I20260927 05:16:22.980540 16379 raft_consensus.cc:348] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.980947 16379 raft_consensus.cc:374] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:22.981038 16379 raft_consensus.cc:729] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf71d7882894d47aa381c553482e363, State: Initialized, Role: FOLLOWER
I20260927 05:16:22.980919 16381 consensus_queue.cc:260] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.981577 16379 consensus_queue.cc:260] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.982419 16380 raft_consensus.cc:348] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.982785 16380 raft_consensus.cc:374] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:22.982875 16380 raft_consensus.cc:729] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbe9c17d0a34386808f4d2c0dbc7c87, State: Initialized, Role: FOLLOWER
I20260927 05:16:22.983366 16379 ts_tablet_manager.cc:1428] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260927 05:16:22.983430 16380 consensus_queue.cc:260] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:22.984278 16228 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:22.984675 16361 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:22.984967 16381 ts_tablet_manager.cc:1428] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260927 05:16:22.987274 16380 ts_tablet_manager.cc:1428] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260927 05:16:23.095863 16095 tablet.cc:2369] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:23.098402 16229 tablet.cc:2369] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:23.110288 16362 tablet.cc:2369] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:16:23.191653 16385 raft_consensus.cc:482] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:16:23.192006 16385 raft_consensus.cc:504] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:23.193735 16385 leader_election.cc:288] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593), adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:23.201375 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.194150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54019 (local address 127.0.97.1:43335)
0927 05:16:23.194353 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.194360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.195166 (+ 806us) server_negotiation.cc:407] Connection header received
0927 05:16:23.195370 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.195375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.195455 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.195556 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:23.196461 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.197625 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.199641 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.200375 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.200557 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.200669 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.200988 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.201111 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:16:23.201416 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.194142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53333 (local address 127.0.97.3:43593)
0927 05:16:23.194353 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.194360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.195185 (+ 825us) server_negotiation.cc:407] Connection header received
0927 05:16:23.195370 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.195375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.195455 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.195556 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:23.196450 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.197625 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.199605 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.200382 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.200533 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.200654 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.201031 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.201155 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:16:23.201561 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.194038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43335 (local address 127.0.97.2:54019)
0927 05:16:23.195031 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:23.195050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:23.195223 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:23.195585 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:23.195590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:23.195621 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:23.196277 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:23.196287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.197765 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.197770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:23.199469 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:23.199482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.200622 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.200629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:23.200806 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.200856 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 05:16:23.200928 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 05:16:23.201067 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":175,"spinlock_wait_cycles":43008,"thread_start_us":516,"threads_started":1}
I20260927 05:16:23.201763 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.194076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43593 (local address 127.0.97.2:53333)
0927 05:16:23.195083 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:23.195100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:23.195223 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:23.195585 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:23.195590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:23.195627 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:23.196251 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:23.196269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.197822 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.197827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:23.199441 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:23.199453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.200658 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.200664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:23.200854 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.200877 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:16:23.200981 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:16:23.201111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":59,"spinlock_wait_cycles":3328,"thread_start_us":585,"threads_started":1}
I20260927 05:16:23.202483 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d03c46899cd46dc9653a443a872e75e" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf71d7882894d47aa381c553482e363" is_pre_election: true
I20260927 05:16:23.202574 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d03c46899cd46dc9653a443a872e75e" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" is_pre_election: true
I20260927 05:16:23.203035 16049 raft_consensus.cc:2454] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 0.
I20260927 05:16:23.203105 16316 raft_consensus.cc:2454] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 0.
I20260927 05:16:23.203706 16119 leader_election.cc:302] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: adf71d7882894d47aa381c553482e363, cfbe9c17d0a34386808f4d2c0dbc7c87, f3626704972a4f33a6797e0a7ea89b8e; no voters:
I20260927 05:16:23.204092 16385 raft_consensus.cc:2781] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:16:23.204232 16385 raft_consensus.cc:482] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:16:23.204332 16385 raft_consensus.cc:3037] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:23.206691 16385 raft_consensus.cc:504] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:23.207444 16385 leader_election.cc:288] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 election: Requested vote from peers cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593), adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:23.207789 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d03c46899cd46dc9653a443a872e75e" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
I20260927 05:16:23.207803 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d03c46899cd46dc9653a443a872e75e" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf71d7882894d47aa381c553482e363"
I20260927 05:16:23.207959 16316 raft_consensus.cc:3037] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:23.207986 16049 raft_consensus.cc:3037] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:23.210045 16316 raft_consensus.cc:2454] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 1.
I20260927 05:16:23.210052 16049 raft_consensus.cc:2454] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 1.
I20260927 05:16:23.210405 16119 leader_election.cc:302] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfbe9c17d0a34386808f4d2c0dbc7c87, f3626704972a4f33a6797e0a7ea89b8e; no voters:
I20260927 05:16:23.210685 16385 raft_consensus.cc:2781] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:16:23.211170 16385 raft_consensus.cc:686] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 1 LEADER]: Becoming Leader. State: Replica: f3626704972a4f33a6797e0a7ea89b8e, State: Running, Role: LEADER
I20260927 05:16:23.211661 16385 consensus_queue.cc:237] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } }
I20260927 05:16:23.216024 15903 catalog_manager.cc:5495] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3626704972a4f33a6797e0a7ea89b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } health_report { overall_health: HEALTHY } } }
I20260927 05:16:23.241180 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.236312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.0.97.2:45199)
0927 05:16:23.236484 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.236488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.236529 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:16:23.236774 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.236778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.236863 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.236980 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:16:23.237312 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.237993 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.239344 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.239745 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.239836 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.240232 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:16:23.240704 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:16:23.240807 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.240900 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.240960 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:16:23.248917 16316 raft_consensus.cc:1261] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Refusing update from remote peer f3626704972a4f33a6797e0a7ea89b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:23.248929 16049 raft_consensus.cc:1261] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Refusing update from remote peer f3626704972a4f33a6797e0a7ea89b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:23.249851 16385 consensus_queue.cc:1035] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:23.250137 16390 consensus_queue.cc:1035] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:23.266086 16393 mvcc.cc:205] Tried to move back new op lower bound from 7333831406578049024 to 7333831406438653952. Current Snapshot: MvccSnapshot[applied={T|T < 7333831406578049024}]
I20260927 05:16:23.267746 16394 mvcc.cc:205] Tried to move back new op lower bound from 7333831406578049024 to 7333831406438653952. Current Snapshot: MvccSnapshot[applied={T|T < 7333831406578049024}]
I20260927 05:16:23.285226 15903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:23.285540 15903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:23.289129 15903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Sending DeleteTablet for 3 replicas of tablet 7d03c46899cd46dc9653a443a872e75e
I20260927 05:16:23.290088 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d03c46899cd46dc9653a443a872e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34788
I20260927 05:16:23.290396 16163 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d03c46899cd46dc9653a443a872e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42328
I20260927 05:16:23.290810 16029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d03c46899cd46dc9653a443a872e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55522
I20260927 05:16:23.291092 16402 tablet_replica.cc:330] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: stopping tablet replica
I20260927 05:16:23.291453 16402 raft_consensus.cc:2229] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:23.291880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15966
I20260927 05:16:23.291905 16402 raft_consensus.cc:2258] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:23.297015 16402 ts_tablet_manager.cc:1905] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:23.301131 16404 tablet_replica.cc:330] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: stopping tablet replica
I20260927 05:16:23.301648 16404 raft_consensus.cc:2229] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:23.302181 16404 raft_consensus.cc:2258] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:23.304036 16404 ts_tablet_manager.cc:1905] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:23.311120 16404 ts_tablet_manager.cc:1918] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:23.311731 16403 tablet_replica.cc:330] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: stopping tablet replica
I20260927 05:16:23.312121 16403 raft_consensus.cc:2229] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:16:23.312773 16403 raft_consensus.cc:2258] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:23.313153 16404 log.cc:1196] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/7d03c46899cd46dc9653a443a872e75e
I20260927 05:16:23.313668 16404 ts_tablet_manager.cc:1939] T 7d03c46899cd46dc9653a443a872e75e P adf71d7882894d47aa381c553482e363: Deleting consensus metadata
I20260927 05:16:23.315317 15890 catalog_manager.cc:4841] TS adf71d7882894d47aa381c553482e363 (127.0.97.1:43335): tablet 7d03c46899cd46dc9653a443a872e75e (table test-table [id=b06e24bb6459496e9c535c5aae043743]) successfully deleted
Sep 27 05:16:23 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:16:23.315575 16402 ts_tablet_manager.cc:1918] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:23.315986 16402 log.cc:1196] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/7d03c46899cd46dc9653a443a872e75e
I20260927 05:16:23.316483 16402 ts_tablet_manager.cc:1939] T 7d03c46899cd46dc9653a443a872e75e P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting consensus metadata
I20260927 05:16:23.317092 16403 ts_tablet_manager.cc:1905] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:23.318148 15892 catalog_manager.cc:4841] TS cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593): tablet 7d03c46899cd46dc9653a443a872e75e (table test-table [id=b06e24bb6459496e9c535c5aae043743]) successfully deleted
I20260927 05:16:23.324404 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.293412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.97.1:43335)
0927 05:16:23.293639 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.293648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.304031 (+ 10383us) server_negotiation.cc:407] Connection header received
0927 05:16:23.304536 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.304557 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.304671 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.304758 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:23.307593 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.309132 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.311735 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.312197 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.312333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.317003 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:23.317016 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:23.317032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:23.317071 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:23.321168 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.322035 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.322043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.322047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.322210 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.322702 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.322708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.322711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.323182 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:23.323464 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.323831 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.324037 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:16:23.324925 16403 ts_tablet_manager.cc:1918] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:23.325124 16403 log.cc:1196] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/7d03c46899cd46dc9653a443a872e75e
I20260927 05:16:23.325127 16089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:23.325686 16403 ts_tablet_manager.cc:1939] T 7d03c46899cd46dc9653a443a872e75e P f3626704972a4f33a6797e0a7ea89b8e: Deleting consensus metadata
I20260927 05:16:23.327380 15891 catalog_manager.cc:4841] TS f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2:45199): tablet 7d03c46899cd46dc9653a443a872e75e (table test-table [id=b06e24bb6459496e9c535c5aae043743]) successfully deleted
I20260927 05:16:23.414101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15966
I20260927 05:16:23.435490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16100
Sep 27 05:16:23 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:16:23.453583 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.436331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.97.2:45199)
0927 05:16:23.436835 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.436841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.436858 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:16:23.436920 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.436925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.437000 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.437092 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:23.438955 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.439908 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.441838 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.442275 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.442363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.447579 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:23.447593 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:23.447608 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:23.447643 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:23.450719 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.451546 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.451553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.451556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.451700 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.452128 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.452132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.452134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.452500 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:23.452760 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.453090 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.453254 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260927 05:16:23.454293 16223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:23.563383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16100
I20260927 05:16:23.582614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16233
Sep 27 05:16:23 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:16:23.597388 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:23.583377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.97.3:43593)
0927 05:16:23.583556 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:23.583560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:23.583648 (+ 88us) server_negotiation.cc:407] Connection header received
0927 05:16:23.583782 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:23.583786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:23.583851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:23.583933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:23.585149 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.585998 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.587799 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:23.588166 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:23.588259 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:23.592549 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:23.592562 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:23.592577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:23.592617 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:23.595160 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.595735 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.595742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.595744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.595866 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:23.596164 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:23.596167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:23.596169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:23.596517 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:23.596733 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:23.596940 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 05:16:23.597083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:16:23.598023 16356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:23.690511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16233
I20260927 05:16:23.708658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 15873
I20260927 05:16:23.709784 15935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:23.817766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 15873
I20260927 05:16:43.839522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--webserver_interface=127.0.97.62
--webserver_port=38489
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:16:44.168465 16410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:44.168799 16410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:44.168957 16410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:44.175956 16410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:16:44.176076 16410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:16:44.176172 16410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:16:44.176265 16410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:16:44.176344 16410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:44.176460 16410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:44.176548 16410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:16:44.176626 16410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:16:44.186026 16410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:44.187332 16410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:44.188588 16410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:44.210742 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:44.210709 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:44.211050 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:44.211622 16410 server_base.cc:1034] running on GCE node
I20260927 05:16:44.212432 16410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:44.213953 16410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:44.215201 16410 hybrid_clock.cc:648] HybridClock initialized: now 1790486204215141 us; error 39 us; skew 500 ppm
I20260927 05:16:44.215577 16410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:44 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:44.223114 16410 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:16:44.224740 16410 webserver.cc:460] Webserver started at http://127.0.97.62:38489/ using document root <none> and password file <none>
I20260927 05:16:44.225503 16410 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:44.225656 16410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:44.231483 16410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:16:44.234265 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:44.235757 16410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:44.235940 16410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "fc71c246477f4b14a9c822cae7167adc"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "3a8aa291d71f057a96e2b79dcb0226fa"
server_key_iv: "361e775a6621ff602cb313de3316d585"
server_key_version: "encryptionkey@0"
I20260927 05:16:44.236967 16410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:44.281350 16410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:44.285683 16410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:44.286019 16410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:44.305356 16410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40863
I20260927 05:16:44.305392 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40863 every 8 connection(s)
I20260927 05:16:44.306911 16410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:16:44.311903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16410
I20260927 05:16:44.312716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43335
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=40481
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:16:44.313968 16478 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:16:44.317198 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap starting.
I20260927 05:16:44.344238 16478 log.cc:824] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:44.358978 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:16:44.359676 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap complete.
I20260927 05:16:44.373335 16478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:44.374449 16478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Initialized, Role: FOLLOWER
I20260927 05:16:44.375187 16478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:44.375445 16478 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:16:44.375587 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:16:44.375782 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:16:44.379630 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:44.380151 16478 leader_election.cc:302] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc71c246477f4b14a9c822cae7167adc; no voters:
I20260927 05:16:44.380832 16478 leader_election.cc:288] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:16:44.380964 16482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:16:44.382082 16482 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 LEADER]: Becoming Leader. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Running, Role: LEADER
I20260927 05:16:44.382696 16482 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:16:44.383116 16478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:16:44.385033 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc71c246477f4b14a9c822cae7167adc. Latest consensus state: current_term: 2 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:16:44.385382 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:44.394086 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:16:44.399367 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:16:44.400233 16487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:16:44.412791 16487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:16:44.414575 16487 catalog_manager.cc:1259] Loaded cluster ID: c093d34e7894455c8a73e0764c34e5b7
I20260927 05:16:44.414758 16487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:16:44.420087 16487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:16:44.422152 16487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Loaded TSK: 0
I20260927 05:16:44.447227 16487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Generated new TSK 1
I20260927 05:16:44.447577 16487 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:16:44.691058 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:44.691330 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:44.691478 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:44.698572 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:44.698671 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:44.698829 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:16:44.708453 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40481
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:44.709623 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:44.710897 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:44.724310 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:44.724543 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:44.725476 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:44.725888 16480 server_base.cc:1034] running on GCE node
I20260927 05:16:44.726619 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:44.727844 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:44.729103 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790486204729060 us; error 54 us; skew 500 ppm
I20260927 05:16:44.729485 16480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:44 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:44.733433 16480 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:16:44.735277 16480 webserver.cc:460] Webserver started at http://127.0.97.1:40481/ using document root <none> and password file <none>
I20260927 05:16:44.736155 16480 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:44.736308 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:44.743103 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.004s
I20260927 05:16:44.746335 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:44.747969 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:16:44.748145 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "adf71d7882894d47aa381c553482e363"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "5463f538e851b603fbb3e184d908c243"
server_key_iv: "eec71e4df392eab81e7cdfa5d509fa52"
server_key_version: "encryptionkey@0"
I20260927 05:16:44.749491 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:44.773592 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:44.778606 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:44.778885 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:44.780009 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:44.781641 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:44.781767 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:44.781875 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:44.781950 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:44.821684 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43335
I20260927 05:16:44.821728 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43335 every 8 connection(s)
I20260927 05:16:44.823397 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:16:44.829098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16480
I20260927 05:16:44.829849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45199
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=39205
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:16:44 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:44.848408 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:44.826797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:36651)
0927 05:16:44.827618 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:44.827663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:44.829435 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:44.830903 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:44.830916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:44.831572 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:44.832085 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:44.832102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:44.833799 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:44.833808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:44.835231 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:44.835242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:44.836261 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:44.836268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:44.836553 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:44.837261 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:44.837299 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:44.839918 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:44.844555 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:44.844561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:44.844571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:44.844956 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:44.845454 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:44.845457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:44.845459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:44.845671 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:44.846594 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:44.846600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:44.846871 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 05:16:44.847130 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 05:16:44.847484 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":356,"threads_started":1}
I20260927 05:16:44.849103 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:44.826502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36651 (local address 127.0.97.62:40863)
0927 05:16:44.827303 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:44.827321 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:44.827785 (+ 464us) server_negotiation.cc:407] Connection header received
0927 05:16:44.829986 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:44.830010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:44.830387 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:44.830759 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:44.832323 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:44.833627 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:44.835367 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:44.836026 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:44.836227 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:44.840226 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:44.840240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:44.840261 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:44.840310 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:44.844424 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:44.845122 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:44.845133 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:44.845143 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:44.845350 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:44.845808 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:44.845816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:44.845830 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:44.846461 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:44.846664 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:44.847403 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 05:16:44.847825 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260927 05:16:44.852088 16628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:44.852437 16628 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:44.853420 16628 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:44.855901 16443 ts_manager.cc:194] Registered new tserver with Master: adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:44.857295 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36651
W20260927 05:16:45.151935 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:45.152204 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:45.152367 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:45.159584 16632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:45.159708 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:45.159884 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:16:45.169090 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39205
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:45.170287 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:45.171615 16632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:45.184528 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:45.184602 16638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:45.184646 16640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:45.185343 16632 server_base.cc:1034] running on GCE node
I20260927 05:16:45.185990 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:45.186892 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:45.188091 16632 hybrid_clock.cc:648] HybridClock initialized: now 1790486205188044 us; error 46 us; skew 500 ppm
I20260927 05:16:45.188395 16632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:45 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:45.191479 16632 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:16:45.192740 16632 webserver.cc:460] Webserver started at http://127.0.97.2:39205/ using document root <none> and password file <none>
I20260927 05:16:45.193320 16632 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:45.193424 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:45.198494 16632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:16:45.201045 16647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.202224 16632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:16:45.202384 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f3626704972a4f33a6797e0a7ea89b8e"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "257d8798a639da52f459282e4a48e224"
server_key_iv: "f3f9fde00e1d8ab9ab0b7dddfdcda358"
server_key_version: "encryptionkey@0"
I20260927 05:16:45.203423 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:45.222477 16632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:45.226261 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:45.226518 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:45.227473 16632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:45.228732 16632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:45.228847 16632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.228991 16632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:45.229077 16632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.260936 16632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45199
I20260927 05:16:45.261018 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45199 every 8 connection(s)
I20260927 05:16:45.262341 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:16:45.265400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16632
I20260927 05:16:45.266023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43593
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=45279
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:16:45 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:45.283078 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.264834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41113 (local address 127.0.97.62:40863)
0927 05:16:45.265072 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.265078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.266218 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 05:16:45.268123 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.268136 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.268225 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.268335 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:45.269760 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.270464 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.272065 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.272470 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.272609 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.276225 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:45.276234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:45.276241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:45.276285 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:45.279860 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.280561 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.280570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.280576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.280740 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.281117 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.281124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.281130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.281518 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:45.281674 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.282276 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.282547 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:16:45.283656 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.265261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:41113)
0927 05:16:45.266064 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:45.266114 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:45.267864 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:45.268567 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:45.268576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:45.269060 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:45.269586 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.269602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.270613 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.270620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.271906 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.271918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.272721 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.272729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.272940 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.273600 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:45.273625 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:45.275980 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:45.280005 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:45.280011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:45.280022 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:45.280428 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:45.280843 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:45.280846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:45.280848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:45.281003 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:45.281637 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:45.281644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:45.281923 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 05:16:45.282139 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 05:16:45.282477 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":364,"threads_started":1}
I20260927 05:16:45.285087 16760 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:45.285389 16760 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:45.286134 16760 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:45.287567 16443 ts_manager.cc:194] Registered new tserver with Master: f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2:45199)
I20260927 05:16:45.288337 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41113
W20260927 05:16:45.583658 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:16:45.583947 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:16:45.584100 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:16:45.591239 16763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:16:45.591347 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:16:45.591526 16763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:16:45.600921 16763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45279
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:16:45.602058 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:16:45.603451 16763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:16:45.616315 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:45.616362 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:16:45.616385 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:16:45.617657 16763 server_base.cc:1034] running on GCE node
I20260927 05:16:45.618202 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:16:45.619094 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:16:45.620280 16763 hybrid_clock.cc:648] HybridClock initialized: now 1790486205620249 us; error 30 us; skew 500 ppm
I20260927 05:16:45.620522 16763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:16:45 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:16:45.623304 16763 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:16:45.624645 16763 webserver.cc:460] Webserver started at http://127.0.97.3:45279/ using document root <none> and password file <none>
I20260927 05:16:45.625320 16763 fs_manager.cc:359] Metadata directory not provided
I20260927 05:16:45.625463 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:16:45.630366 16763 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:16:45.632874 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.634055 16763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 05:16:45.634217 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "ac0ce6af33491cc7b2e4847ca1cfe2e1"
server_key_iv: "3f250499a6c73df58f496a0b6cc2d79f"
server_key_version: "encryptionkey@0"
I20260927 05:16:45.635118 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:16:45.662015 16763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:16:45.667209 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:16:45.667510 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:16:45.668848 16763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:16:45.670406 16763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:16:45.670492 16763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.670567 16763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:16:45.670621 16763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:16:45.703151 16763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43593
I20260927 05:16:45.703280 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43593 every 8 connection(s)
I20260927 05:16:45.704734 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:16:45.709009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16763
Sep 27 05:16:45 dist-test-slave-1t6p krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:16:45.727612 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.707203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43823 (local address 127.0.97.62:40863)
0927 05:16:45.707402 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.707407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.708508 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 05:16:45.710937 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.710942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.711047 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.711140 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:45.713491 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.714191 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.716073 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.716770 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.716901 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.720744 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:45.720749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:45.720752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:45.720781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:45.724153 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.725062 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.725068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.725073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.725218 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.725766 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.725769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.725772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.726094 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:45.726241 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.727203 (+ 962us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.727372 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:16:45.728940 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.707712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:43823)
0927 05:16:45.708364 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:45.708403 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:45.710606 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:45.711468 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:45.711483 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:45.712384 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:16:45.713298 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.713326 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.714333 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.714340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.715946 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.715957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.716959 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.716963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.717165 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.717880 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:16:45.717910 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:16:45.720520 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:16:45.724304 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:45.724314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:45.724330 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:45.724939 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:45.725336 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:16:45.725342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:16:45.725347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:16:45.725613 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:16:45.726261 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:16:45.726272 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:16:45.726702 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 05:16:45.727080 (+ 378us) client_negotiation.cc:240] Negotiation successful
0927 05:16:45.727681 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":250,"threads_started":1}
I20260927 05:16:45.730355 16891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40863
I20260927 05:16:45.730629 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.709944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.0.97.62:40863)
0927 05:16:45.710407 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.710411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.715379 (+ 4968us) server_negotiation.cc:407] Connection header received
0927 05:16:45.715558 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.715569 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.715619 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.715730 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:45.717721 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.718852 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.720625 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.721116 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.721205 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.724365 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:45.724373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:45.724381 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:45.724417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:45.727850 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.728472 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.728477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.728480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.728630 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.728986 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.728990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.728993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.729322 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:45.729578 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.730078 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.730276 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260927 05:16:45.730664 16891 heartbeater.cc:464] Registering TS with master...
I20260927 05:16:45.731499 16891 heartbeater.cc:511] Master 127.0.97.62:40863 requested a full tablet report, sending...
I20260927 05:16:45.733098 16443 ts_manager.cc:194] Registered new tserver with Master: cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593)
I20260927 05:16:45.733888 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43823
I20260927 05:16:45.745256 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:16:45.751448 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.746191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.0.97.62:40863)
0927 05:16:45.746361 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.746365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.746797 (+ 432us) server_negotiation.cc:407] Connection header received
0927 05:16:45.746960 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.746964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.747168 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.747286 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:16:45.747644 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.748304 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.749711 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.750113 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.750209 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.750716 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:16:45.750906 (+ 190us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 05:16:45.751262 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60014: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 05:16:45.751581 16894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60014: token expired
W20260927 05:16:45.752132 16365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:40863)
I20260927 05:16:45.764238 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.752572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.97.62:40863)
0927 05:16:45.752738 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.752742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.752757 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:16:45.752873 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.752877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.752927 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.752999 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:45.754352 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.755127 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.756473 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.756871 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.756959 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.759459 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:45.759464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:45.759467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:45.759495 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:45.762061 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.762631 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.762635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.762638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.762755 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:45.763123 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:45.763127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:45.763129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:45.763408 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:45.763585 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.763797 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.764017 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:16:45.768281 16443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:16:45.775525 16443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:16:45.809595 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.787550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43593 (local address 127.0.0.1:41582)
0927 05:16:45.797712 (+ 10162us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:45.797754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:45.797921 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:45.801796 (+ 3875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:45.801807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:45.801847 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:45.802338 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.802352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.804302 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.804308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.806636 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.806682 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.808624 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.808630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.808845 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.808896 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 05:16:45.809040 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:16:45.809226 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9916,"thread_start_us":12723,"threads_started":1}
I20260927 05:16:45.809594 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.788227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.97.3:43593)
0927 05:16:45.801018 (+ 12791us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.801031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.801072 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:16:45.801194 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.801202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.801530 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.801782 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:45.802600 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.804169 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.806838 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.808385 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.808542 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.808785 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.809071 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.809229 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12571,"thread_start_us":12454,"threads_started":1}
I20260927 05:16:45.811592 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.787805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45199 (local address 127.0.0.1:59798)
0927 05:16:45.800550 (+ 12745us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:45.800571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:45.800689 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:45.803798 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:45.803803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:45.803827 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:45.804205 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.804223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.806123 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.806128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.808418 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.808432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.810786 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.810792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.810969 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.811001 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:16:45.811099 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:16:45.811235 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12618,"thread_start_us":16609,"threads_started":1}
I20260927 05:16:45.811905 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.788770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.97.2:45199)
0927 05:16:45.803126 (+ 14356us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.803138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.803171 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:16:45.803274 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.803284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.803562 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.803783 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:45.804445 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.805981 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.808591 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.810362 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.810477 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.810648 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.811405 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.811581 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14138,"thread_start_us":13970,"threads_started":1}
I20260927 05:16:45.811975 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.788123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43335 (local address 127.0.0.1:55606)
0927 05:16:45.804391 (+ 16268us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:45.804408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:45.804525 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:45.805078 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:45.805082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:45.805104 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:45.805476 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.805484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.807253 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.807258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.809404 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:45.809418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.811213 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.811220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:45.811398 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.811428 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:16:45.811516 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:16:45.811647 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16135,"thread_start_us":9223,"threads_started":1}
I20260927 05:16:45.812922 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:45.789212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.1:43335)
0927 05:16:45.802586 (+ 13374us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:45.802598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:45.804655 (+ 2057us) server_negotiation.cc:407] Connection header received
0927 05:16:45.804762 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:45.804767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:45.804906 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:45.805090 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:45.805639 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.807086 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.809581 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:45.810987 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:45.811130 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:45.811299 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:45.812500 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0927 05:16:45.812652 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13164,"thread_start_us":13050,"threads_started":1}
I20260927 05:16:45.813727 16826 tablet_service.cc:1467] Processing CreateTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 (DEFAULT_TABLE table=test-table [id=44f548e7a7d84053ba6e000556613f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:45.814965 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a299ad2776c4f0d94451f6d64ad1b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:45.815227 16695 tablet_service.cc:1467] Processing CreateTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 (DEFAULT_TABLE table=test-table [id=44f548e7a7d84053ba6e000556613f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:45.816504 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a299ad2776c4f0d94451f6d64ad1b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:45.816120 16563 tablet_service.cc:1467] Processing CreateTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 (DEFAULT_TABLE table=test-table [id=44f548e7a7d84053ba6e000556613f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:16:45.817378 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a299ad2776c4f0d94451f6d64ad1b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:16:45.834189 16907 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Bootstrap starting.
I20260927 05:16:45.837116 16908 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Bootstrap starting.
I20260927 05:16:45.837700 16907 tablet_bootstrap.cc:654] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:45.839154 16907 log.cc:824] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:45.839820 16909 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Bootstrap starting.
I20260927 05:16:45.841413 16908 tablet_bootstrap.cc:654] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:45.842386 16908 log.cc:824] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:45.844026 16909 tablet_bootstrap.cc:654] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Neither blocks nor log segments found. Creating new log.
I20260927 05:16:45.845188 16909 log.cc:824] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Log is configured to *not* fsync() on all Append() calls
I20260927 05:16:45.845471 16908 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: No bootstrap required, opened a new log
I20260927 05:16:45.845623 16907 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: No bootstrap required, opened a new log
I20260927 05:16:45.845798 16908 ts_tablet_manager.cc:1397] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:16:45.845889 16907 ts_tablet_manager.cc:1397] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 05:16:45.848173 16909 tablet_bootstrap.cc:492] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: No bootstrap required, opened a new log
I20260927 05:16:45.848413 16909 ts_tablet_manager.cc:1397] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 05:16:45.853057 16907 raft_consensus.cc:348] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.853442 16907 raft_consensus.cc:374] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:45.853545 16907 raft_consensus.cc:729] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbe9c17d0a34386808f4d2c0dbc7c87, State: Initialized, Role: FOLLOWER
I20260927 05:16:45.854164 16907 consensus_queue.cc:260] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.855325 16891 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:45.856287 16907 ts_tablet_manager.cc:1428] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260927 05:16:45.856290 16908 raft_consensus.cc:348] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.856740 16908 raft_consensus.cc:374] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:45.856845 16908 raft_consensus.cc:729] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3626704972a4f33a6797e0a7ea89b8e, State: Initialized, Role: FOLLOWER
I20260927 05:16:45.857399 16908 consensus_queue.cc:260] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.859123 16909 raft_consensus.cc:348] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.859647 16909 raft_consensus.cc:374] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:16:45.859773 16909 raft_consensus.cc:729] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf71d7882894d47aa381c553482e363, State: Initialized, Role: FOLLOWER
I20260927 05:16:45.860646 16628 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:45.860531 16909 consensus_queue.cc:260] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:45.861120 16908 ts_tablet_manager.cc:1428] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Time spent starting tablet: real 0.015s user 0.008s sys 0.010s
I20260927 05:16:45.861771 16760 heartbeater.cc:503] Master 127.0.97.62:40863 was elected leader, sending a full tablet report...
I20260927 05:16:45.862773 16909 ts_tablet_manager.cc:1428] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260927 05:16:45.956327 16892 tablet.cc:2369] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:46.014281 16761 tablet.cc:2369] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:16:46.076033 16629 tablet.cc:2369] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:16:46.125298 16914 raft_consensus.cc:482] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:16:46.125559 16914 raft_consensus.cc:504] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:46.127252 16914 leader_election.cc:288] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593), adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:46.133790 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.127628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34173 (local address 127.0.97.3:43593)
0927 05:16:46.127865 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.127870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.128594 (+ 724us) server_negotiation.cc:407] Connection header received
0927 05:16:46.128816 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.128821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.128880 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.128967 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:46.129579 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.130273 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.132101 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.132871 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.133013 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.133115 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.133431 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.133543 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:16:46.133914 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.127756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49151 (local address 127.0.97.1:43335)
0927 05:16:46.127962 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.127971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.128647 (+ 676us) server_negotiation.cc:407] Connection header received
0927 05:16:46.128845 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.128851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.128914 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.129010 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:16:46.129592 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.130277 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.132109 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.133003 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.133164 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.133262 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.133547 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.133660 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:16:46.133929 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.127458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43593 (local address 127.0.97.2:34173)
0927 05:16:46.128475 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:46.128491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:46.128657 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:46.128994 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:46.128999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:46.129031 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:46.129424 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:46.129433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.130396 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.130403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:46.131952 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:46.131964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.133068 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.133074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:46.133241 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.133288 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 05:16:46.133392 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:16:46.133507 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":89,"thread_start_us":565,"threads_started":1}
I20260927 05:16:46.134040 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.127458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43335 (local address 127.0.97.2:49151)
0927 05:16:46.128514 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 05:16:46.128529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:16:46.128713 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:16:46.129024 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:16:46.129028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:16:46.129049 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:16:46.129439 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:46.129448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.130408 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.130414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:46.131961 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:16:46.131972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.133235 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.133241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:16:46.133401 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.133421 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 05:16:46.133514 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:16:46.133638 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":77,"thread_start_us":543,"threads_started":1}
I20260927 05:16:46.134934 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a299ad2776c4f0d94451f6d64ad1b11" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf71d7882894d47aa381c553482e363" is_pre_election: true
I20260927 05:16:46.135041 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a299ad2776c4f0d94451f6d64ad1b11" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" is_pre_election: true
I20260927 05:16:46.135468 16846 raft_consensus.cc:2454] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 0.
I20260927 05:16:46.135468 16583 raft_consensus.cc:2454] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 0.
I20260927 05:16:46.136080 16649 leader_election.cc:302] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf71d7882894d47aa381c553482e363, f3626704972a4f33a6797e0a7ea89b8e; no voters:
I20260927 05:16:46.136448 16914 raft_consensus.cc:2781] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:16:46.136597 16914 raft_consensus.cc:482] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:16:46.136710 16914 raft_consensus.cc:3037] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:46.139035 16914 raft_consensus.cc:504] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:46.139621 16914 leader_election.cc:288] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 election: Requested vote from peers cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593), adf71d7882894d47aa381c553482e363 (127.0.97.1:43335)
I20260927 05:16:46.140036 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a299ad2776c4f0d94451f6d64ad1b11" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
I20260927 05:16:46.140097 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a299ad2776c4f0d94451f6d64ad1b11" candidate_uuid: "f3626704972a4f33a6797e0a7ea89b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf71d7882894d47aa381c553482e363"
I20260927 05:16:46.140249 16846 raft_consensus.cc:3037] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:46.140300 16583 raft_consensus.cc:3037] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:16:46.143402 16846 raft_consensus.cc:2454] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 1.
I20260927 05:16:46.143402 16583 raft_consensus.cc:2454] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3626704972a4f33a6797e0a7ea89b8e in term 1.
I20260927 05:16:46.143778 16649 leader_election.cc:302] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf71d7882894d47aa381c553482e363, f3626704972a4f33a6797e0a7ea89b8e; no voters:
I20260927 05:16:46.144115 16914 raft_consensus.cc:2781] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:16:46.144482 16914 raft_consensus.cc:686] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 1 LEADER]: Becoming Leader. State: Replica: f3626704972a4f33a6797e0a7ea89b8e, State: Running, Role: LEADER
I20260927 05:16:46.144896 16914 consensus_queue.cc:237] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } }
I20260927 05:16:46.149207 16442 catalog_manager.cc:5495] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3626704972a4f33a6797e0a7ea89b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3626704972a4f33a6797e0a7ea89b8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 } health_report { overall_health: UNKNOWN } } }
I20260927 05:16:46.169669 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.164363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.0.97.2:45199)
0927 05:16:46.164577 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.164586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.164606 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:16:46.164835 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.164839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.164921 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.165069 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:16:46.165440 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.166160 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.167606 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.167998 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.168125 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.168624 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:16:46.169110 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:16:46.169262 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.169338 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.169414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:16:46.177301 16846 raft_consensus.cc:1261] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Refusing update from remote peer f3626704972a4f33a6797e0a7ea89b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:46.177307 16583 raft_consensus.cc:1261] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Refusing update from remote peer f3626704972a4f33a6797e0a7ea89b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:16:46.178242 16918 consensus_queue.cc:1035] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:46.178534 16914 consensus_queue.cc:1035] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf71d7882894d47aa381c553482e363" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:16:46.194422 16921 mvcc.cc:205] Tried to move back new op lower bound from 7333831500492922880 to 7333831500372758528. Current Snapshot: MvccSnapshot[applied={T|T < 7333831500492922880}]
I20260927 05:16:46.208280 16442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:46.208564 16442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:16:46.212088 16442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Sending DeleteTablet for 3 replicas of tablet 1a299ad2776c4f0d94451f6d64ad1b11
I20260927 05:16:46.213150 16826 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41582
I20260927 05:16:46.213160 16695 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59798
I20260927 05:16:46.213825 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a299ad2776c4f0d94451f6d64ad1b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55606
I20260927 05:16:46.214033 16932 tablet_replica.cc:330] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: stopping tablet replica
I20260927 05:16:46.214414 16932 raft_consensus.cc:2229] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:16:46.214601 16933 tablet_replica.cc:330] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: stopping tablet replica
I20260927 05:16:46.214814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16480
I20260927 05:16:46.214968 16933 raft_consensus.cc:2229] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:46.215036 16932 raft_consensus.cc:2258] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:46.215466 16933 raft_consensus.cc:2258] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:46.215763 16931 tablet_replica.cc:330] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: stopping tablet replica
I20260927 05:16:46.216140 16931 raft_consensus.cc:2229] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:16:46.216576 16931 raft_consensus.cc:2258] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:16:46.217692 16933 ts_tablet_manager.cc:1905] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:46.218534 16932 ts_tablet_manager.cc:1905] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:46.219429 16931 ts_tablet_manager.cc:1905] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:16:46.225790 16932 ts_tablet_manager.cc:1918] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:46.225996 16932 log.cc:1196] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/1a299ad2776c4f0d94451f6d64ad1b11
I20260927 05:16:46.226560 16932 ts_tablet_manager.cc:1939] T 1a299ad2776c4f0d94451f6d64ad1b11 P f3626704972a4f33a6797e0a7ea89b8e: Deleting consensus metadata
I20260927 05:16:46.228348 16429 catalog_manager.cc:4841] TS f3626704972a4f33a6797e0a7ea89b8e (127.0.97.2:45199): tablet 1a299ad2776c4f0d94451f6d64ad1b11 (table test-table [id=44f548e7a7d84053ba6e000556613f3f]) successfully deleted
I20260927 05:16:46.229173 16931 ts_tablet_manager.cc:1918] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:46.229346 16931 log.cc:1196] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/1a299ad2776c4f0d94451f6d64ad1b11
I20260927 05:16:46.229923 16933 ts_tablet_manager.cc:1918] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:16:46.230011 16931 ts_tablet_manager.cc:1939] T 1a299ad2776c4f0d94451f6d64ad1b11 P cfbe9c17d0a34386808f4d2c0dbc7c87: Deleting consensus metadata
I20260927 05:16:46.230122 16933 log.cc:1196] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/1a299ad2776c4f0d94451f6d64ad1b11
I20260927 05:16:46.230665 16933 ts_tablet_manager.cc:1939] T 1a299ad2776c4f0d94451f6d64ad1b11 P adf71d7882894d47aa381c553482e363: Deleting consensus metadata
I20260927 05:16:46.231992 16430 catalog_manager.cc:4841] TS cfbe9c17d0a34386808f4d2c0dbc7c87 (127.0.97.3:43593): tablet 1a299ad2776c4f0d94451f6d64ad1b11 (table test-table [id=44f548e7a7d84053ba6e000556613f3f]) successfully deleted
I20260927 05:16:46.232448 16428 catalog_manager.cc:4841] TS adf71d7882894d47aa381c553482e363 (127.0.97.1:43335): tablet 1a299ad2776c4f0d94451f6d64ad1b11 (table test-table [id=44f548e7a7d84053ba6e000556613f3f]) successfully deleted
I20260927 05:16:46.240509 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.216566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.0.97.1:43335)
0927 05:16:46.216733 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.216739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.216756 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:16:46.216812 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.216819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.216900 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.217010 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:46.218936 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.220048 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.221924 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.222556 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.222663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.232868 (+ 10205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:46.232879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:46.232891 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:46.232934 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:46.236506 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.237592 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.237602 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.237606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.237762 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.238223 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.238229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.238232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.238707 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:46.238992 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.239846 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.240068 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:16:46.241369 16623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:46.355257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16480
I20260927 05:16:46.372949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16632
I20260927 05:16:46.388566 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.373850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.0.97.2:45199)
0927 05:16:46.374052 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.374058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.374080 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:16:46.374147 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.374153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.374230 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.374352 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:46.375677 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.376898 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.378205 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.378903 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.379052 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.381743 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:46.381758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:46.381783 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:46.381829 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:46.385711 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.386419 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.386428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.386432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.386598 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.386942 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.386948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.386951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.387441 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:46.387773 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.387939 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.388156 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:16:46.389472 16755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:46.500702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16632
I20260927 05:16:46.521107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16763
I20260927 05:16:46.534152 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:16:46.521857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.97.3:43593)
0927 05:16:46.522079 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:16:46.522084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:16:46.522104 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:16:46.522269 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:16:46.522274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:16:46.522337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:16:46.522471 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:16:46.523820 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.524543 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.525802 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:16:46.526188 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:16:46.526316 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:16:46.528809 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:16:46.528822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:16:46.528837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:16:46.528869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:16:46.531462 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.532154 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.532161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.532163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.532294 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:16:46.532670 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:16:46.532680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:16:46.532681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:16:46.533006 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:16:46.533300 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:16:46.533593 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 05:16:46.533821 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:16:46.534832 16886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:46.623092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16763
I20260927 05:16:46.641417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 16410
I20260927 05:16:46.642519 16473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:16:46.755635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 16410
I20260927 05:17:06.778806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--webserver_interface=127.0.97.62
--webserver_port=38489
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:07.099501 16938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:07.099763 16938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:07.099864 16938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:07.106777 16938 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:17:07.106865 16938 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:17:07.106904 16938 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:17:07.106954 16938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:07.107051 16938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:07.107136 16938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:07.107182 16938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:07.107235 16938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:07.116619 16938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:07.117743 16938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:07.118938 16938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:07.130620 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:07.130625 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:07.130980 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:07.131423 16938 server_base.cc:1034] running on GCE node
I20260927 05:17:07.132117 16938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:07.133457 16938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:07.134706 16938 hybrid_clock.cc:648] HybridClock initialized: now 1790486227134644 us; error 34 us; skew 500 ppm
I20260927 05:17:07.135085 16938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:07 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:07.138018 16938 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:17:07.139479 16938 webserver.cc:460] Webserver started at http://127.0.97.62:38489/ using document root <none> and password file <none>
I20260927 05:17:07.140147 16938 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:07.140275 16938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:07.145352 16938 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:17:07.147962 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:07.149281 16938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:07.149458 16938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "fc71c246477f4b14a9c822cae7167adc"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "3a8aa291d71f057a96e2b79dcb0226fa"
server_key_iv: "361e775a6621ff602cb313de3316d585"
server_key_version: "encryptionkey@0"
I20260927 05:17:07.150388 16938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:07.173594 16938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:07.177390 16938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:07.177682 16938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:07.196197 16938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40863
I20260927 05:17:07.196241 17004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40863 every 8 connection(s)
I20260927 05:17:07.197665 16938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:07.199985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 16938
I20260927 05:17:07.200975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43335
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=40481
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:17:07.206737 17005 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:17:07.211113 17005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap starting.
I20260927 05:17:07.230945 17005 log.cc:824] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:07.246002 17005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:17:07.246526 17005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Bootstrap complete.
I20260927 05:17:07.256116 17005 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:17:07.257050 17005 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Initialized, Role: FOLLOWER
I20260927 05:17:07.257661 17005 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:17:07.257905 17005 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:07.258028 17005 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:07.258204 17005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:17:07.261891 17005 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:17:07.262354 17005 leader_election.cc:302] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc71c246477f4b14a9c822cae7167adc; no voters:
I20260927 05:17:07.263042 17005 leader_election.cc:288] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:17:07.263178 17009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:17:07.264221 17009 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [term 3 LEADER]: Becoming Leader. State: Replica: fc71c246477f4b14a9c822cae7167adc, State: Running, Role: LEADER
I20260927 05:17:07.264798 17009 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } }
I20260927 05:17:07.265157 17005 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:07.267411 17011 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc71c246477f4b14a9c822cae7167adc. Latest consensus state: current_term: 3 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:17:07.267834 17011 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:07.268424 17010 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fc71c246477f4b14a9c822cae7167adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc71c246477f4b14a9c822cae7167adc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40863 } } }
I20260927 05:17:07.268769 17010 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:07.270872 17015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:07.279664 17015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:07.281076 17015 catalog_manager.cc:1259] Loaded cluster ID: c093d34e7894455c8a73e0764c34e5b7
I20260927 05:17:07.281220 17015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:07.285156 17015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:07.286836 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Loaded TSK: 0
I20260927 05:17:07.287063 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc71c246477f4b14a9c822cae7167adc: Loaded TSK: 1
F20260927 05:17:07.288251 17015 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f870b1189ad google::LogMessage::Fail() at ??:0
@ 0x7f870b11b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f870b1184de google::LogMessage::Flush() at ??:0
@ 0x7f870b119a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f870ee7c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f870ee6b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8720b471c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8720b46783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8720bd22a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f8720bd20c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f8720c1c5e9 std::function<>::operator()() at ??:0
@ 0x7f8720b4acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8720b48c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8720bd0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8720bd0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f870dd64345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f870dd6791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f870dd67799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f870dd42eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f871101c6db start_thread at ??:0
@ 0x7f8709fb261f clone at ??:0
W20260927 05:17:07.604774 17007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:07.605023 17007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:07.605185 17007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:07.612190 17007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:07.612280 17007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:07.612430 17007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:07.621604 17007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40481
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:07.622778 17007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:07.624078 17007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:07.636986 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:07.637038 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:07.636999 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:07.637792 17007 server_base.cc:1034] running on GCE node
I20260927 05:17:07.638360 17007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:07.639269 17007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:07.640462 17007 hybrid_clock.cc:648] HybridClock initialized: now 1790486227640424 us; error 35 us; skew 500 ppm
I20260927 05:17:07.640763 17007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:07 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:07.643698 17007 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:17:07.645071 17007 webserver.cc:460] Webserver started at http://127.0.97.1:40481/ using document root <none> and password file <none>
I20260927 05:17:07.645714 17007 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:07.645843 17007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:07.650800 17007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:07.653344 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:07.654567 17007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:17:07.654731 17007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "adf71d7882894d47aa381c553482e363"
format_stamp: "Formatted at 2026-09-27 05:16:21 on dist-test-slave-1t6p"
server_key: "5463f538e851b603fbb3e184d908c243"
server_key_iv: "eec71e4df392eab81e7cdfa5d509fa52"
server_key_version: "encryptionkey@0"
I20260927 05:17:07.655742 17007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:07.690934 17007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:07.694659 17007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:07.694871 17007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:07.695802 17007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:07.696979 17007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:07.697063 17007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:07.697149 17007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:07.697206 17007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:07.729413 17007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43335
I20260927 05:17:07.729492 17174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43335 every 8 connection(s)
I20260927 05:17:07.730819 17007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:07.735390 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:07.733623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:60909)
0927 05:17:07.734320 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:07.734510 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
W20260927 05:17:07.736017 17175 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40863 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40863: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
I20260927 05:17:07.736487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17007
I20260927 05:17:07.736815 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:07.736505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:42761)
0927 05:17:07.736649 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:07.736720 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:17:07.737175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45199
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=39205
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:17:07.737619 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:07.737264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:45239)
0927 05:17:07.737451 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:07.737516 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 05:17:07.737835 17175 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:17:08.055694 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:08.055946 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:08.056146 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:08.063401 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:08.063520 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:08.063704 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:08.073242 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39205
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:08.074419 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:08.075696 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:08.088703 17183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:08.088747 17186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:08.088775 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:08.089671 17178 server_base.cc:1034] running on GCE node
I20260927 05:17:08.090225 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:08.091142 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:08.092360 17178 hybrid_clock.cc:648] HybridClock initialized: now 1790486228092301 us; error 47 us; skew 500 ppm
I20260927 05:17:08.092651 17178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:08 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:08.095616 17178 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:17:08.096917 17178 webserver.cc:460] Webserver started at http://127.0.97.2:39205/ using document root <none> and password file <none>
I20260927 05:17:08.097584 17178 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:08.097729 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:08.102665 17178 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:17:08.105192 17193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:08.106426 17178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:17:08.106596 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f3626704972a4f33a6797e0a7ea89b8e"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "257d8798a639da52f459282e4a48e224"
server_key_iv: "f3f9fde00e1d8ab9ab0b7dddfdcda358"
server_key_version: "encryptionkey@0"
I20260927 05:17:08.107618 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:08.126641 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:08.130239 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:08.130498 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:08.131474 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:08.132737 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:08.132860 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:08.132995 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:08.133091 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:08.165913 17178 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45199
I20260927 05:17:08.165995 17305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45199 every 8 connection(s)
I20260927 05:17:08.167431 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:08.168839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17178
I20260927 05:17:08.169446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43593
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40863
--webserver_port=45279
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:17:08.172339 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.170268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:42743)
0927 05:17:08.171113 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.171348 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":348,"threads_started":1}
W20260927 05:17:08.173008 17306 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40863 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40863: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
I20260927 05:17:08.173828 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.173524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:42463)
0927 05:17:08.173659 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.173722 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:17:08.174597 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.174294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:57601)
0927 05:17:08.174428 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.174499 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:17:08.174808 17306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:17:08.483356 17309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:08.483645 17309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:08.483809 17309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:08.490900 17309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:08.491070 17309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:08.491247 17309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:08.500576 17309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45279
--tserver_master_addrs=127.0.97.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:08.501801 17309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:08.503105 17309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:08.516567 17315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:08.516661 17314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:08.517803 17317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:08.518672 17309 server_base.cc:1034] running on GCE node
I20260927 05:17:08.519434 17309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:08.520612 17309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:08.521823 17309 hybrid_clock.cc:648] HybridClock initialized: now 1790486228521776 us; error 39 us; skew 500 ppm
I20260927 05:17:08.522162 17309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:08 dist-test-slave-1t6p krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:08.525142 17309 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:17:08.526397 17309 webserver.cc:460] Webserver started at http://127.0.97.3:45279/ using document root <none> and password file <none>
I20260927 05:17:08.527058 17309 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:08.527165 17309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:08.532043 17309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 05:17:08.534403 17324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:08.535660 17309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 05:17:08.535836 17309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "cfbe9c17d0a34386808f4d2c0dbc7c87"
format_stamp: "Formatted at 2026-09-27 05:16:22 on dist-test-slave-1t6p"
server_key: "ac0ce6af33491cc7b2e4847ca1cfe2e1"
server_key_iv: "3f250499a6c73df58f496a0b6cc2d79f"
server_key_version: "encryptionkey@0"
I20260927 05:17:08.536746 17309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:08.552299 17309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:08.556350 17309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:08.556617 17309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:08.557524 17309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:08.558735 17309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:08.558884 17309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:08.558995 17309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:08.559088 17309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 05:17:08.592579 17309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43593
I20260927 05:17:08.592670 17436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43593 every 8 connection(s)
I20260927 05:17:08.593977 17309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:08.598557 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.596782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:34765)
0927 05:17:08.597512 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.597709 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
W20260927 05:17:08.599267 17437 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40863 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40863: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
I20260927 05:17:08.600090 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.599710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:51407)
0927 05:17:08.599840 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.599934 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 05:17:08.600348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17309
I20260927 05:17:08.600967 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.600600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:48435)
0927 05:17:08.600747 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.600838 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:17:08.601205 17437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:17:08.739161 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:08.738372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:42961)
0927 05:17:08.738777 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:08.738883 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260927 05:17:09.175988 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:09.175309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:55609)
0927 05:17:09.175702 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:09.175779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":192,"threads_started":1}
I20260927 05:17:09.602375 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:09.601661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:37189)
0927 05:17:09.602059 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:09.602140 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":194,"threads_started":1}
I20260927 05:17:09.740872 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:09.739976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:41117)
0927 05:17:09.740517 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:09.740625 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260927 05:17:10.177469 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:10.176812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:60285)
0927 05:17:10.177226 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:10.177310 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260927 05:17:10.603888 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:10.603224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:35121)
0927 05:17:10.603647 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:10.603734 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260927 05:17:10.742470 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:10.741743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:45255)
0927 05:17:10.742218 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:10.742305 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":253,"threads_started":1}
I20260927 05:17:11.179207 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:11.178493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:46541)
0927 05:17:11.178928 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:11.179040 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260927 05:17:11.605396 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:11.604723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:58231)
0927 05:17:11.605151 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:11.605235 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260927 05:17:11.743999 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:11.743354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:54555)
0927 05:17:11.743754 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:11.743837 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":206,"threads_started":1}
I20260927 05:17:12.180742 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:12.180063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:58107)
0927 05:17:12.180482 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:12.180555 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260927 05:17:12.606911 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:12.606249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:44183)
0927 05:17:12.606652 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:12.606731 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":199,"threads_started":1}
I20260927 05:17:12.745566 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:12.744819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:35545)
0927 05:17:12.745317 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:12.745397 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260927 05:17:13.182284 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:13.181554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:45323)
0927 05:17:13.182047 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:13.182120 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":261,"threads_started":1}
I20260927 05:17:13.608625 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:13.607811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:35571)
0927 05:17:13.608333 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:13.608414 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":256,"threads_started":1}
I20260927 05:17:13.747212 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:13.746447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:40267)
0927 05:17:13.746872 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:13.746956 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260927 05:17:14.183938 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:14.183262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:38185)
0927 05:17:14.183693 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:14.183777 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260927 05:17:14.610222 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:14.609530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:34201)
0927 05:17:14.609962 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:14.610047 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260927 05:17:14.748944 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:14.748145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:55431)
0927 05:17:14.748653 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:14.748739 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":215,"threads_started":1}
I20260927 05:17:15.185518 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:15.184755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:60231)
0927 05:17:15.185228 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:15.185305 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":215,"threads_started":1}
I20260927 05:17:15.611907 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:15.611098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:34177)
0927 05:17:15.611621 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:15.611695 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":272,"threads_started":1}
I20260927 05:17:15.750566 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:15.749874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:53205)
0927 05:17:15.750318 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:15.750399 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260927 05:17:16.187505 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:16.186735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:39407)
0927 05:17:16.187276 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:16.187349 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":324,"threads_started":1}
I20260927 05:17:16.613525 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:16.612799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:46715)
0927 05:17:16.613307 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:16.613380 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":282,"threads_started":1}
I20260927 05:17:16.752197 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:16.751500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:41621)
0927 05:17:16.751954 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:16.752031 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":222,"threads_started":1}
I20260927 05:17:17.189040 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:17.188360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:51099)
0927 05:17:17.188788 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:17.188860 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":208,"threads_started":1}
I20260927 05:17:17.615108 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:17.614376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:56075)
0927 05:17:17.614810 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:17.614883 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":211,"threads_started":1}
I20260927 05:17:17.753656 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:17.753036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:51587)
0927 05:17:17.753431 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:17.753504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":217,"threads_started":1}
I20260927 05:17:18.190555 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:18.189892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:50667)
0927 05:17:18.190328 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:18.190401 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260927 05:17:18.616518 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:18.615892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:39931)
0927 05:17:18.616284 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:18.616355 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":213,"threads_started":1}
I20260927 05:17:18.755285 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:18.754510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:47705)
0927 05:17:18.754937 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:18.755105 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260927 05:17:19.192080 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:19.191411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:49101)
0927 05:17:19.191832 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:19.191913 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260927 05:17:19.618211 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:19.617538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:33493)
0927 05:17:19.617966 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:19.618047 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260927 05:17:19.756889 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:19.756177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:56079)
0927 05:17:19.756657 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:19.756734 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260927 05:17:20.193621 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:20.192941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:56249)
0927 05:17:20.193392 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:20.193466 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260927 05:17:20.619652 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:20.619074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:58327)
0927 05:17:20.619445 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:20.619515 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":187,"threads_started":1}
I20260927 05:17:20.758531 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:20.757822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:47517)
0927 05:17:20.758269 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:20.758351 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":225,"threads_started":1}
I20260927 05:17:21.195381 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:21.194586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:43419)
0927 05:17:21.195096 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:21.195178 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260927 05:17:21.621161 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:21.620446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:52587)
0927 05:17:21.620899 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:21.620980 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":224,"threads_started":1}
I20260927 05:17:21.760468 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:21.759604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:59619)
0927 05:17:21.760120 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:21.760243 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
I20260927 05:17:22.196991 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:22.196278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:45095)
0927 05:17:22.196737 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:22.196814 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260927 05:17:22.622654 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:22.622013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:41709)
0927 05:17:22.622413 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:22.622486 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260927 05:17:22.762199 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:22.761518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.1:49427)
0927 05:17:22.761940 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:22.762021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":233,"threads_started":1}
I20260927 05:17:23.198745 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:23.197999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.2:40495)
0927 05:17:23.198477 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:23.198559 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260927 05:17:23.607285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17007
I20260927 05:17:23.624529 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:23.623775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40863 (local address 127.0.97.3:40083)
0927 05:17:23.624276 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:23.624363 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":275,"threads_started":1}
I20260927 05:17:23.628018 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:23.608311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.0.97.1:43335)
0927 05:17:23.608969 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:23.608993 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:23.609033 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:17:23.610960 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:23.610969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:23.611499 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:23.611805 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:23.613182 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.614368 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.615663 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.616321 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.616460 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:23.622211 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:23.622219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:23.622234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:23.622271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:23.625182 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.625834 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.625840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.625845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.625996 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.626351 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.626354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.626356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.626848 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:23.627132 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:23.627447 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 05:17:23.627774 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":258,"threads_started":1}
I20260927 05:17:23.629350 17170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:23.716148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17007
I20260927 05:17:23.732890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17178
I20260927 05:17:23.750672 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:23.733845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.0.97.2:45199)
0927 05:17:23.734390 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:23.734404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:23.734440 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:17:23.736078 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:23.736088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:23.736517 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:23.736889 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:23.738297 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.739733 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.741058 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.741765 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.741949 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:23.744609 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:23.744618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:23.744634 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:23.744670 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:23.747523 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.748182 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.748189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.748195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.748354 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.748740 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.748744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.748746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.749229 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:23.749527 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:23.749977 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 05:17:23.750376 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260927 05:17:23.752023 17301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:23.834189 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17178
I20260927 05:17:23.851125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17309
I20260927 05:17:23.868474 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:23.852065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.97.3:43593)
0927 05:17:23.852663 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:23.852677 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:23.852718 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:17:23.854288 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:23.854297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:23.854722 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:23.855135 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:23.856459 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.857717 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.858981 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:23.859722 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:23.859857 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:23.862272 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:23.862280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:23.862295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:23.862329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:23.865503 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.866100 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.866108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.866115 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.866282 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:23.866635 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:23.866638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:23.866640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:23.867202 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:23.867426 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:23.867780 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 05:17:23.868183 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260927 05:17:23.869760 17432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:23.954211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17309
2026-09-27T05:17:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64437 ms)
[----------] 30 tests from SecurityITest (292872 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:24 dist-test-slave-1t6p krb5kdc[17494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:24 dist-test-slave-1t6p krb5kdc[17494](info): set up 2 sockets
Sep 27 05:17:24 dist-test-slave-1t6p krb5kdc[17494](info): commencing operation
krb5kdc: starting...
W20260927 05:17:25.054936 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:25 dist-test-slave-1t6p krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:17:25.280660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38483
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:25.603886 17507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:25.604149 17507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:25.604313 17507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:25.611469 17507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:25.611565 17507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:25.611651 17507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:25.611734 17507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:25.611827 17507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:25.621310 17507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:25.622565 17507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:25.623916 17507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:25.635932 17515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:25.636006 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:25.635978 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:25.636427 17507 server_base.cc:1034] running on GCE node
I20260927 05:17:25.637188 17507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:25.638477 17507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:25.639755 17507 hybrid_clock.cc:648] HybridClock initialized: now 1790486245639729 us; error 94 us; skew 500 ppm
I20260927 05:17:25.640082 17507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:25 dist-test-slave-1t6p krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:25.643072 17507 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:17:25.644345 17507 webserver.cc:460] Webserver started at http://127.0.97.62:38653/ using document root <none> and password file <none>
I20260927 05:17:25.644939 17507 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:25.645032 17507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:25.645277 17507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:25.648061 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "a4311051017a4acc959780ecfb6c4021"
format_stamp: "Formatted at 2026-09-27 05:17:25 on dist-test-slave-1t6p"
server_key: "0bf761253fde9840b0768a3de48a500a"
server_key_iv: "93367e58d45635f783a7c6acf79ef5f6"
server_key_version: "encryptionkey@0"
I20260927 05:17:25.648718 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "a4311051017a4acc959780ecfb6c4021"
format_stamp: "Formatted at 2026-09-27 05:17:25 on dist-test-slave-1t6p"
server_key: "0bf761253fde9840b0768a3de48a500a"
server_key_iv: "93367e58d45635f783a7c6acf79ef5f6"
server_key_version: "encryptionkey@0"
I20260927 05:17:25.653388 17507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:25.656490 17522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:25.657802 17507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:17:25.657929 17507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "a4311051017a4acc959780ecfb6c4021"
format_stamp: "Formatted at 2026-09-27 05:17:25 on dist-test-slave-1t6p"
server_key: "0bf761253fde9840b0768a3de48a500a"
server_key_iv: "93367e58d45635f783a7c6acf79ef5f6"
server_key_version: "encryptionkey@0"
I20260927 05:17:25.658083 17507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:25.672704 17507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:25.676416 17507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:25.676633 17507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:25.694303 17507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38483
I20260927 05:17:25.694352 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38483 every 8 connection(s)
I20260927 05:17:25.696025 17507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:25.699606 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:25.701656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17507
I20260927 05:17:25.701876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:25.702708 388 external_mini_cluster.cc:1428] Setting key 21dd4b0f15f4b26a9a5ca017cea07a20
I20260927 05:17:25.712148 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: Bootstrap starting.
I20260927 05:17:25.715862 17574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:25.717129 17574 log.cc:824] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:17:25 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:25.720031 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: No bootstrap required, opened a new log
I20260927 05:17:25.725862 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:25.704720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.0.97.62:38483)
0927 05:17:25.705517 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:25.705528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:25.705568 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:17:25.707296 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:25.707317 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:25.707661 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:25.707954 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:25.709738 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:25.711071 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:25.712898 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:25.713520 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:25.713651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:25.718760 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:25.718767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:25.718781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:25.718813 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:25.721881 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:25.722626 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:25.722637 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:25.722648 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:25.722823 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:25.723239 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:25.723259 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:25.723261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:25.723736 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:25.724043 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:25.724492 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 05:17:25.724896 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":371,"threads_started":1}
I20260927 05:17:25.728183 17574 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260927 05:17:25.728607 17574 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:25.728716 17574 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4311051017a4acc959780ecfb6c4021, State: Initialized, Role: FOLLOWER
I20260927 05:17:25.729290 17574 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260927 05:17:25.729491 17574 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:25.729580 17574 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:25.729692 17574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:25.732146 17574 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260927 05:17:25.732553 17574 leader_election.cc:302] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4311051017a4acc959780ecfb6c4021; no voters:
I20260927 05:17:25.733076 17574 leader_election.cc:288] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:25.733168 17579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:25.733834 17579 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [term 1 LEADER]: Becoming Leader. State: Replica: a4311051017a4acc959780ecfb6c4021, State: Running, Role: LEADER
I20260927 05:17:25.734380 17579 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260927 05:17:25.735025 17574 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:25.737090 17581 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4311051017a4acc959780ecfb6c4021. Latest consensus state: current_term: 1 leader_uuid: "a4311051017a4acc959780ecfb6c4021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } } }
I20260927 05:17:25.737032 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4311051017a4acc959780ecfb6c4021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4311051017a4acc959780ecfb6c4021" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } } }
I20260927 05:17:25.737448 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:25.737448 17581 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:25.738307 17586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:25.743180 17586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:25.753690 17586 catalog_manager.cc:1347] Generated new cluster ID: 52d1b8a59cdb43268ce2dc7ede209589
I20260927 05:17:25.753804 17586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:25.763620 17586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:25.765336 17586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:25.773058 17586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: Generated new TSK 0
I20260927 05:17:25.773917 17586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:17:25.830371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:26.154623 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:26.154918 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:26.155107 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:26.162006 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:26.162144 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:26.162331 17600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:26.171926 17600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:26.173223 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:26.174484 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:26.187186 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:26.187175 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:26.187288 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:26.188370 17600 server_base.cc:1034] running on GCE node
I20260927 05:17:26.188997 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:26.189890 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:26.191103 17600 hybrid_clock.cc:648] HybridClock initialized: now 1790486246191071 us; error 51 us; skew 500 ppm
I20260927 05:17:26.191406 17600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:26 dist-test-slave-1t6p krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:26.194476 17600 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:17:26.195961 17600 webserver.cc:460] Webserver started at http://127.0.97.1:45523/ using document root <none> and password file <none>
I20260927 05:17:26.196602 17600 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:26.196753 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:26.197048 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:26.199844 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8933218c59a4396aa37835ecad27936"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "bdffb12b38c3697ee575fac6f695055e"
server_key_iv: "ab1389ad5883caeacdf79269cc49d0ec"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.200591 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8933218c59a4396aa37835ecad27936"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "bdffb12b38c3697ee575fac6f695055e"
server_key_iv: "ab1389ad5883caeacdf79269cc49d0ec"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.205355 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 05:17:26.208628 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.209926 17600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:17:26.210160 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "e8933218c59a4396aa37835ecad27936"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "bdffb12b38c3697ee575fac6f695055e"
server_key_iv: "ab1389ad5883caeacdf79269cc49d0ec"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.210379 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:26.226135 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:26.230264 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:26.230599 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:26.231601 17600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:26.232795 17600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:26.232873 17600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.232957 17600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:26.233011 17600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.265751 17600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42773
I20260927 05:17:26.265838 17727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42773 every 8 connection(s)
I20260927 05:17:26.266978 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:26.274477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17600
I20260927 05:17:26.274676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:26.275473 388 external_mini_cluster.cc:1428] Setting key 97d59b0112e94354cf5fd0ecdcbf2f74
Sep 27 05:17:26 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:26.288383 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:26.269402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57951 (local address 127.0.97.62:38483)
0927 05:17:26.269845 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:26.269849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:26.270660 (+ 811us) server_negotiation.cc:407] Connection header received
0927 05:17:26.272522 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:26.272528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:26.272638 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:26.272762 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:26.274251 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.275483 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.277519 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.278140 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.278284 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:26.281619 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:26.281628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:26.281635 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:26.281678 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:26.285209 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:26.285901 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:26.285908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:26.285912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:26.286038 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:26.286446 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:26.286452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:26.286455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:26.286709 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:26.286876 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:26.287649 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 05:17:26.287891 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":249,"threads_started":1}
I20260927 05:17:26.288992 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:26.269860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.1:57951)
0927 05:17:26.270510 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:26.270552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:26.272256 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:26.272960 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:26.272968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:26.273541 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:26.274075 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:26.274091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.275666 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.275670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:26.277376 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:26.277390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.278333 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.278337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:26.278518 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:26.279178 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:26.279200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:26.281414 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:26.285345 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:26.285352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:26.285362 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:26.285769 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:26.286149 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:26.286154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:26.286158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:26.286338 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:26.286927 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:26.286933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:26.287292 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 05:17:26.287522 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 05:17:26.287872 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":263,"threads_started":1}
I20260927 05:17:26.290592 17728 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260927 05:17:26.290894 17728 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:26.291590 17728 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260927 05:17:26.293810 17539 ts_manager.cc:194] Registered new tserver with Master: e8933218c59a4396aa37835ecad27936 (127.0.97.1:42773)
I20260927 05:17:26.295951 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57951
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:17:26.331302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:26.670081 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:26.670318 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:26.670408 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:26.677265 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:26.677366 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:26.677757 17734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:26.687424 17734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:26.688535 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:26.689720 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:26.708279 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:26.708365 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:26.708300 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:26.709218 17734 server_base.cc:1034] running on GCE node
I20260927 05:17:26.709863 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:26.710935 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:26.712185 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790486246712128 us; error 44 us; skew 500 ppm
I20260927 05:17:26.712558 17734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:26 dist-test-slave-1t6p krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:26.716066 17734 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:17:26.717492 17734 webserver.cc:460] Webserver started at http://127.0.97.2:40067/ using document root <none> and password file <none>
I20260927 05:17:26.718145 17734 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:26.718294 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:26.718595 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:26.721412 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "35a661af0d834e4994af2c7ae51e96c4"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "1cac032fd3fcd9cd3fafe00f32cf65c6"
server_key_iv: "e565c74d2b410a1703631a167311169a"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.722190 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "35a661af0d834e4994af2c7ae51e96c4"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "1cac032fd3fcd9cd3fafe00f32cf65c6"
server_key_iv: "e565c74d2b410a1703631a167311169a"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.726985 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 05:17:26.730293 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.731600 17734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:17:26.731767 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "35a661af0d834e4994af2c7ae51e96c4"
format_stamp: "Formatted at 2026-09-27 05:17:26 on dist-test-slave-1t6p"
server_key: "1cac032fd3fcd9cd3fafe00f32cf65c6"
server_key_iv: "e565c74d2b410a1703631a167311169a"
server_key_version: "encryptionkey@0"
I20260927 05:17:26.731951 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:26.744867 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:26.748883 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:26.749179 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:26.750062 17734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:26.751374 17734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:26.751444 17734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.751605 17734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:26.751660 17734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:26.784622 17734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34927
I20260927 05:17:26.784667 17861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34927 every 8 connection(s)
I20260927 05:17:26.785884 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:26.793426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17734
I20260927 05:17:26.793617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:26.794135 388 external_mini_cluster.cc:1428] Setting key 36862905f9d6f3e71585ca2518e54fec
Sep 27 05:17:26 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:26.806648 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:26.788388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59867 (local address 127.0.97.62:38483)
0927 05:17:26.788571 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:26.788575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:26.789726 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 05:17:26.791483 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:26.791487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:26.791554 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:26.791642 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:26.792987 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.794120 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.796118 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.796760 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.796887 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:26.800420 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:26.800424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:26.800428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:26.800456 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:26.803505 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:26.804216 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:26.804222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:26.804226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:26.804389 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:26.804792 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:26.804799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:26.804805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:26.805186 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:26.805336 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:26.805946 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 05:17:26.806206 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:17:26.807248 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:26.788866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.2:59867)
0927 05:17:26.789562 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:26.789609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:26.791243 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:26.791835 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:26.791843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:26.792325 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:26.792853 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:26.792867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.794314 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.794318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:26.795966 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:26.795980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:26.796941 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:26.796946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:26.797122 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:26.797781 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:26.797801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:26.800216 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:26.803666 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:26.803672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:26.803683 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:26.804077 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:26.804499 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:26.804502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:26.804504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:26.804674 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:26.805302 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:26.805308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:26.805584 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 05:17:26.805807 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 05:17:26.806160 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":294,"threads_started":1}
I20260927 05:17:26.808678 17862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260927 05:17:26.808938 17862 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:26.809544 17862 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260927 05:17:26.810935 17539 ts_manager.cc:194] Registered new tserver with Master: 35a661af0d834e4994af2c7ae51e96c4 (127.0.97.2:34927)
I20260927 05:17:26.811735 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59867
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:17:26.846302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:27.165743 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:27.165999 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:27.166173 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:27.173172 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:27.173269 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:27.173475 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:27.182592 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:27.183835 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:27.185098 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:27.198591 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:27.198539 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:27.198635 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:27.199538 17867 server_base.cc:1034] running on GCE node
I20260927 05:17:27.200162 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:27.201002 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:27.202206 17867 hybrid_clock.cc:648] HybridClock initialized: now 1790486247202172 us; error 34 us; skew 500 ppm
I20260927 05:17:27.202528 17867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:27 dist-test-slave-1t6p krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:27.205601 17867 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:17:27.206926 17867 webserver.cc:460] Webserver started at http://127.0.97.3:36119/ using document root <none> and password file <none>
I20260927 05:17:27.207612 17867 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:27.207738 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:27.208072 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:27.210808 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9"
format_stamp: "Formatted at 2026-09-27 05:17:27 on dist-test-slave-1t6p"
server_key: "434b8bbb0eb4a11315e2e59d3c2ce256"
server_key_iv: "9c9f77dc72e638eb4563728691b209dd"
server_key_version: "encryptionkey@0"
I20260927 05:17:27.211722 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9"
format_stamp: "Formatted at 2026-09-27 05:17:27 on dist-test-slave-1t6p"
server_key: "434b8bbb0eb4a11315e2e59d3c2ce256"
server_key_iv: "9c9f77dc72e638eb4563728691b209dd"
server_key_version: "encryptionkey@0"
I20260927 05:17:27.216832 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:17:27.220247 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:27.221556 17867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:17:27.221745 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9"
format_stamp: "Formatted at 2026-09-27 05:17:27 on dist-test-slave-1t6p"
server_key: "434b8bbb0eb4a11315e2e59d3c2ce256"
server_key_iv: "9c9f77dc72e638eb4563728691b209dd"
server_key_version: "encryptionkey@0"
I20260927 05:17:27.221947 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:27.237373 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:27.241636 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:27.241916 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:27.242831 17867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:27.244133 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:27.244211 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:27.244382 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:27.244433 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:27.276394 17867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34839
I20260927 05:17:27.276414 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34839 every 8 connection(s)
I20260927 05:17:27.277644 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:27.286727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 17867
I20260927 05:17:27.286916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:27.287569 388 external_mini_cluster.cc:1428] Setting key 6961a191249e8b393fc8cfb71606c87c
Sep 27 05:17:27 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:27.297040 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.280104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38007 (local address 127.0.97.62:38483)
0927 05:17:27.280324 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.280331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.281395 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 05:17:27.283121 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.283125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.283187 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.283277 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:27.284782 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.285531 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.287253 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.287871 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.288016 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.291810 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:27.291815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:27.291818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:27.291852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:27.294230 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.294980 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.294985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.294988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.295175 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.295575 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.295578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.295580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.295841 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:27.295962 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.296641 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.296806 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:17:27.298026 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.280561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.3:38007)
0927 05:17:27.281241 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.281284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.282861 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.283526 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.283535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.284080 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:27.284615 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.284632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.285712 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.285716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.287105 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.287117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.288109 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.288119 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.288421 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.289154 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:27.289188 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:27.291629 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:27.294384 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:27.294391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:27.294402 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:27.294839 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:27.295307 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:27.295310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:27.295312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:27.295479 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:27.296002 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:27.296008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:27.296309 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.296527 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.296877 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":283,"threads_started":1}
I20260927 05:17:27.298874 17728 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
I20260927 05:17:27.299572 17995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260927 05:17:27.299852 17995 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:27.300751 17995 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260927 05:17:27.302307 17539 ts_manager.cc:194] Registered new tserver with Master: 2c6df80ad8d34875a1e8e620b4dfc5a9 (127.0.97.3:34839)
I20260927 05:17:27.303083 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:27.303346 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38007
I20260927 05:17:27.320299 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.308414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.0.97.62:38483)
0927 05:17:27.308562 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.308565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.309043 (+ 478us) server_negotiation.cc:407] Connection header received
0927 05:17:27.309162 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.309166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.309217 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.309293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:27.310584 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.311388 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.312647 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.313015 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.313100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.315554 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:27.315559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:27.315562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:27.315590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:27.318066 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.318647 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.318652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.318656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.318759 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.319101 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.319104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.319106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.319387 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:27.319561 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.319821 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.319967 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:17:27.324083 17539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:27.330843 17539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:27.354923 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.342529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48210)
0927 05:17:27.343319 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.343365 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.343531 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.344572 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.344583 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.344632 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:27.345117 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.345133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.347644 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.347652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.350750 (+ 3098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.350769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.353626 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.353635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.353913 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.353991 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.354144 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.354399 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":355,"threads_started":1}
I20260927 05:17:27.356277 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.342846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.97.1:42773)
0927 05:17:27.343510 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.343519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.343549 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:17:27.343758 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.343765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.344021 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.344204 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:27.345363 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.347450 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.350992 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.353011 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.353168 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.353477 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.355583 (+ 2106us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.355818 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":317,"threads_started":1}
I20260927 05:17:27.365386 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 88a08900641f4925960769e6356a5b7e (DEFAULT_TABLE table=test-table [id=e0fe79429f8c4bc4a4c6969b949e4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:27.366513 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.345689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.0.97.2:34927)
0927 05:17:27.359576 (+ 13887us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.359589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.359625 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:17:27.359742 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.359751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.360060 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.360314 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:27.360825 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.362306 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.364068 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.365465 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.365619 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.365820 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.366030 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.366232 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13673,"thread_start_us":13538,"threads_started":1}
I20260927 05:17:27.366513 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.345434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34927 (local address 127.0.0.1:43214)
0927 05:17:27.355234 (+ 9800us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.355269 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.355390 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.360311 (+ 4921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.360319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.360357 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:27.360626 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.360640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.362456 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.362462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.363895 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.363925 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.365632 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.365638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.365819 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.365885 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.365992 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.366142 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9661,"mutex_wait_us":3,"thread_start_us":15268,"threads_started":1}
I20260927 05:17:27.366920 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a08900641f4925960769e6356a5b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:27.369282 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.343628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34839 (local address 127.0.0.1:51972)
0927 05:17:27.344809 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.344826 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.344938 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.361921 (+ 16983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.361925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.361949 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:27.362435 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.362449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.364447 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.364452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.366567 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.366581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.368593 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.368605 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.368718 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.368739 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.368841 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.368958 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":913,"threads_started":1}
I20260927 05:17:27.369653 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.344136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.97.3:34839)
0927 05:17:27.361416 (+ 17280us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.361426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.361455 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:17:27.361544 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.361553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.361748 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.361929 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:27.362646 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.364287 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.367070 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.368145 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.368285 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.368511 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.368893 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.369039 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17108,"thread_start_us":16943,"threads_started":1}
I20260927 05:17:27.370100 17797 tablet_service.cc:1467] Processing CreateTablet for tablet 88a08900641f4925960769e6356a5b7e (DEFAULT_TABLE table=test-table [id=e0fe79429f8c4bc4a4c6969b949e4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:27.371500 17797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a08900641f4925960769e6356a5b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:27.372491 17930 tablet_service.cc:1467] Processing CreateTablet for tablet 88a08900641f4925960769e6356a5b7e (DEFAULT_TABLE table=test-table [id=e0fe79429f8c4bc4a4c6969b949e4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:27.373503 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a08900641f4925960769e6356a5b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:27.384284 18013 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Bootstrap starting.
I20260927 05:17:27.386135 18014 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Bootstrap starting.
I20260927 05:17:27.388310 18013 tablet_bootstrap.cc:654] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:27.389097 18015 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Bootstrap starting.
I20260927 05:17:27.389621 18013 log.cc:824] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:27.390056 18014 tablet_bootstrap.cc:654] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:27.390962 18014 log.cc:824] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:27.392551 18015 tablet_bootstrap.cc:654] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:27.393661 18015 log.cc:824] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:27.393798 18013 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: No bootstrap required, opened a new log
I20260927 05:17:27.394045 18013 ts_tablet_manager.cc:1397] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 05:17:27.396641 18014 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: No bootstrap required, opened a new log
I20260927 05:17:27.396634 18015 tablet_bootstrap.cc:492] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: No bootstrap required, opened a new log
I20260927 05:17:27.396881 18014 ts_tablet_manager.cc:1397] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260927 05:17:27.396880 18015 ts_tablet_manager.cc:1397] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:17:27.401612 18013 raft_consensus.cc:348] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.401954 18013 raft_consensus.cc:374] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:27.402077 18013 raft_consensus.cc:729] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8933218c59a4396aa37835ecad27936, State: Initialized, Role: FOLLOWER
I20260927 05:17:27.402639 18013 consensus_queue.cc:260] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.404296 18013 ts_tablet_manager.cc:1428] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 05:17:27.407385 18014 raft_consensus.cc:348] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.407385 18015 raft_consensus.cc:348] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.407783 18015 raft_consensus.cc:374] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:27.407783 18014 raft_consensus.cc:374] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:27.407884 18014 raft_consensus.cc:729] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6df80ad8d34875a1e8e620b4dfc5a9, State: Initialized, Role: FOLLOWER
I20260927 05:17:27.407892 18015 raft_consensus.cc:729] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a661af0d834e4994af2c7ae51e96c4, State: Initialized, Role: FOLLOWER
I20260927 05:17:27.408432 18014 consensus_queue.cc:260] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.408432 18015 consensus_queue.cc:260] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.409889 17995 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
I20260927 05:17:27.409916 17862 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
I20260927 05:17:27.410148 18015 ts_tablet_manager.cc:1428] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 05:17:27.410465 18014 ts_tablet_manager.cc:1428] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 05:17:27.482275 18020 raft_consensus.cc:482] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:27.482597 18020 raft_consensus.cc:504] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.484206 18020 leader_election.cc:288] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8933218c59a4396aa37835ecad27936 (127.0.97.1:42773), 2c6df80ad8d34875a1e8e620b4dfc5a9 (127.0.97.3:34839)
I20260927 05:17:27.492050 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.484699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34839 (local address 127.0.97.2:60763)
0927 05:17:27.485728 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.485747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.485894 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.486227 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.486232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.486259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:27.486732 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.486741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.487938 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.487944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.489989 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.490003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.491157 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.491163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.491325 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.491348 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.491450 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.491574 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":28,"spinlock_wait_cycles":20352,"thread_start_us":547,"threads_started":1}
I20260927 05:17:27.492540 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.484721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60763 (local address 127.0.97.3:34839)
0927 05:17:27.484916 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.484921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.485846 (+ 925us) server_negotiation.cc:407] Connection header received
0927 05:17:27.486038 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.486045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.486110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.486214 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:27.486886 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.487785 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.490111 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.490927 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.491098 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.491206 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.491485 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.491609 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:17:27.492830 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.484620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58277 (local address 127.0.97.1:42773)
0927 05:17:27.484824 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.484830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.485744 (+ 914us) server_negotiation.cc:407] Connection header received
0927 05:17:27.485974 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.485983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.486045 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.486141 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:27.486713 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.487473 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.490084 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.491582 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.491738 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.491856 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.492458 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.492599 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:17:27.493062 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.484697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.97.2:58277)
0927 05:17:27.485632 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.485648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:27.485836 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:27.486158 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:27.486163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:27.486186 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:27.486569 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.486581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.487598 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.487604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.489895 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:27.489911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.492034 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.492046 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:27.492271 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.492328 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 05:17:27.492429 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:17:27.492581 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":4,"spinlock_wait_cycles":25472,"thread_start_us":550,"threads_started":1}
I20260927 05:17:27.493662 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a08900641f4925960769e6356a5b7e" candidate_uuid: "35a661af0d834e4994af2c7ae51e96c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" is_pre_election: true
I20260927 05:17:27.494235 17950 raft_consensus.cc:2454] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a661af0d834e4994af2c7ae51e96c4 in term 0.
I20260927 05:17:27.494244 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a08900641f4925960769e6356a5b7e" candidate_uuid: "35a661af0d834e4994af2c7ae51e96c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8933218c59a4396aa37835ecad27936" is_pre_election: true
I20260927 05:17:27.494751 17683 raft_consensus.cc:2454] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a661af0d834e4994af2c7ae51e96c4 in term 0.
I20260927 05:17:27.494911 17751 leader_election.cc:302] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6df80ad8d34875a1e8e620b4dfc5a9, 35a661af0d834e4994af2c7ae51e96c4; no voters:
I20260927 05:17:27.495384 18020 raft_consensus.cc:2781] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:27.495525 18020 raft_consensus.cc:482] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:27.495635 18020 raft_consensus.cc:3037] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:27.497782 18020 raft_consensus.cc:504] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.498363 18020 leader_election.cc:288] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [CANDIDATE]: Term 1 election: Requested vote from peers e8933218c59a4396aa37835ecad27936 (127.0.97.1:42773), 2c6df80ad8d34875a1e8e620b4dfc5a9 (127.0.97.3:34839)
I20260927 05:17:27.498773 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a08900641f4925960769e6356a5b7e" candidate_uuid: "35a661af0d834e4994af2c7ae51e96c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8933218c59a4396aa37835ecad27936"
I20260927 05:17:27.498819 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a08900641f4925960769e6356a5b7e" candidate_uuid: "35a661af0d834e4994af2c7ae51e96c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9"
I20260927 05:17:27.498960 17683 raft_consensus.cc:3037] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:27.499063 17950 raft_consensus.cc:3037] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:27.501151 17683 raft_consensus.cc:2454] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a661af0d834e4994af2c7ae51e96c4 in term 1.
I20260927 05:17:27.501302 17950 raft_consensus.cc:2454] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a661af0d834e4994af2c7ae51e96c4 in term 1.
I20260927 05:17:27.501538 17752 leader_election.cc:302] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a661af0d834e4994af2c7ae51e96c4, e8933218c59a4396aa37835ecad27936; no voters:
I20260927 05:17:27.501824 18020 raft_consensus.cc:2781] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:27.502276 18020 raft_consensus.cc:686] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 1 LEADER]: Becoming Leader. State: Replica: 35a661af0d834e4994af2c7ae51e96c4, State: Running, Role: LEADER
I20260927 05:17:27.502719 18020 consensus_queue.cc:237] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } }
I20260927 05:17:27.507664 17539 catalog_manager.cc:5495] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a661af0d834e4994af2c7ae51e96c4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "35a661af0d834e4994af2c7ae51e96c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a661af0d834e4994af2c7ae51e96c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34927 } health_report { overall_health: HEALTHY } } }
W20260927 05:17:27.519250 17729 tablet.cc:2369] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:27.529222 17996 tablet.cc:2369] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:27.537861 17863 tablet.cc:2369] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:27.555590 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.550651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.97.2:34927)
0927 05:17:27.550841 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.550845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.550932 (+ 87us) server_negotiation.cc:407] Connection header received
0927 05:17:27.551140 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.551144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.551226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.551347 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:27.551663 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.552370 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.553645 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.554049 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.554137 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.554564 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:27.555113 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:27.555236 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.555320 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.555380 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:17:27.563232 17683 raft_consensus.cc:1261] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 1 FOLLOWER]: Refusing update from remote peer 35a661af0d834e4994af2c7ae51e96c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:27.563277 17950 raft_consensus.cc:1261] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 1 FOLLOWER]: Refusing update from remote peer 35a661af0d834e4994af2c7ae51e96c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:27.564203 18024 consensus_queue.cc:1035] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8933218c59a4396aa37835ecad27936" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:27.564558 18020 consensus_queue.cc:1035] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6df80ad8d34875a1e8e620b4dfc5a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:27.574522 18029 mvcc.cc:205] Tried to move back new op lower bound from 7333831670009438208 to 7333831669775073280. Current Snapshot: MvccSnapshot[applied={T|T < 7333831670009438208}]
I20260927 05:17:27.574872 18027 mvcc.cc:205] Tried to move back new op lower bound from 7333831670009438208 to 7333831669775073280. Current Snapshot: MvccSnapshot[applied={T|T < 7333831670009438208}]
I20260927 05:17:27.590477 17539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:27.590762 17539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:27.594201 17539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4311051017a4acc959780ecfb6c4021: Sending DeleteTablet for 3 replicas of tablet 88a08900641f4925960769e6356a5b7e
I20260927 05:17:27.595391 17930 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a08900641f4925960769e6356a5b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51972
I20260927 05:17:27.595664 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a08900641f4925960769e6356a5b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48210
I20260927 05:17:27.596323 17797 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a08900641f4925960769e6356a5b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43214
I20260927 05:17:27.596698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17600
I20260927 05:17:27.606102 18038 tablet_replica.cc:330] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: stopping tablet replica
I20260927 05:17:27.606549 18038 raft_consensus.cc:2229] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:27.607112 18038 raft_consensus.cc:2258] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:27.608764 18039 tablet_replica.cc:330] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: stopping tablet replica
I20260927 05:17:27.609150 18039 raft_consensus.cc:2229] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:27.609844 18039 raft_consensus.cc:2258] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 05:17:27 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:17:27.612238 18039 ts_tablet_manager.cc:1905] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:27.612659 18037 tablet_replica.cc:330] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: stopping tablet replica
I20260927 05:17:27.613058 18037 raft_consensus.cc:2229] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:27.613662 18037 raft_consensus.cc:2258] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:27.615751 18037 ts_tablet_manager.cc:1905] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:27.616329 18038 ts_tablet_manager.cc:1905] T 88a08900641f4925960769e6356a5b7e P e8933218c59a4396aa37835ecad27936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:27.620389 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.597873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.97.1:42773)
0927 05:17:27.598040 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.598047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.599520 (+ 1473us) server_negotiation.cc:407] Connection header received
0927 05:17:27.599814 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.599822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.599923 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.600009 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:27.602269 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.603821 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.606234 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.607265 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.607429 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.613070 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:27.613082 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:27.613096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:27.613139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:27.616972 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.617793 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.617801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.617804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.617968 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.618431 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.618436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.618440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.618859 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:27.619159 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.619738 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.619968 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:17:27.621739 17723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:27.622337 18037 ts_tablet_manager.cc:1918] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:27.622491 18037 log.cc:1196] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/88a08900641f4925960769e6356a5b7e
I20260927 05:17:27.623160 18037 ts_tablet_manager.cc:1939] T 88a08900641f4925960769e6356a5b7e P 2c6df80ad8d34875a1e8e620b4dfc5a9: Deleting consensus metadata
I20260927 05:17:27.625442 17524 catalog_manager.cc:4841] TS 2c6df80ad8d34875a1e8e620b4dfc5a9 (127.0.97.3:34839): tablet 88a08900641f4925960769e6356a5b7e (table test-table [id=e0fe79429f8c4bc4a4c6969b949e4330]) successfully deleted
I20260927 05:17:27.629498 18039 ts_tablet_manager.cc:1918] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:27.629683 18039 log.cc:1196] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/88a08900641f4925960769e6356a5b7e
I20260927 05:17:27.630183 18039 ts_tablet_manager.cc:1939] T 88a08900641f4925960769e6356a5b7e P 35a661af0d834e4994af2c7ae51e96c4: Deleting consensus metadata
I20260927 05:17:27.631594 17525 catalog_manager.cc:4841] TS 35a661af0d834e4994af2c7ae51e96c4 (127.0.97.2:34927): tablet 88a08900641f4925960769e6356a5b7e (table test-table [id=e0fe79429f8c4bc4a4c6969b949e4330]) successfully deleted
I20260927 05:17:27.760419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17600
W20260927 05:17:27.778214 17525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42773 (error 108)
I20260927 05:17:27.778946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17734
I20260927 05:17:27.780342 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.779750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48228)
0927 05:17:27.780018 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.780144 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260927 05:17:27.780616 17525 catalog_manager.cc:4584] TS e8933218c59a4396aa37835ecad27936 (127.0.97.1:42773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88a08900641f4925960769e6356a5b7e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Sep 27 05:17:27 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:17:27.795343 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.779750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.0.97.2:34927)
0927 05:17:27.779976 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.779983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.780076 (+ 93us) server_negotiation.cc:407] Connection header received
0927 05:17:27.780261 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.780267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.780353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.780461 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:27.782159 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.782949 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.784499 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.785003 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.785143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.789347 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:27.789358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:27.789373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:27.789409 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:27.792394 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.793188 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.793196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.793198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.793334 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.793691 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.793699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.793702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.794105 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:27.794424 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.794676 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.794888 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 05:17:27.796100 17857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:27.834384 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.833778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48232)
0927 05:17:27.834072 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.834196 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 05:17:27.835492 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.835099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48234)
0927 05:17:27.835264 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.835349 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 05:17:27.896237 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.895702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48238)
0927 05:17:27.895929 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.896034 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:17:27.897300 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.896937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48244)
0927 05:17:27.897090 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.897172 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:17:27.913249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17734
I20260927 05:17:27.931811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17867
Sep 27 05:17:27 dist-test-slave-1t6p krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:17:27.946822 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.932554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.0.97.3:34839)
0927 05:17:27.932733 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:27.932739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:27.932836 (+ 97us) server_negotiation.cc:407] Connection header received
0927 05:17:27.933034 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:27.933040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:27.933147 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:27.933262 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:27.934567 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.935366 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.936729 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:27.937122 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:27.937256 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:27.941356 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:27.941370 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:27.941387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:27.941417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:27.944047 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.944759 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.944774 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.944776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.944919 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:27.945302 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:27.945309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:27.945312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:27.945660 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:27.945961 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:27.946257 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 05:17:27.946469 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:17:27.947446 17990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:27.978236 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.977687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48250)
0927 05:17:27.977942 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.978043 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 05:17:27.979357 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:27.978931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48254)
0927 05:17:27.979167 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:27.979241 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 05:17:28.034536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17867
I20260927 05:17:28.051167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 17507
I20260927 05:17:28.052289 17569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:28.159040 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:28.158423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42773 (local address 127.0.0.1:48268)
0927 05:17:28.158636 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:28.158767 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:17:28.159174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 17507
2026-09-27T05:17:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T05:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:28Z Disabled control of system clock
I20260927 05:17:28.232942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:28.546233 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:28.546502 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:28.546651 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:28.556459 18049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:28.556581 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:28.556654 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:28.556733 18049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:28.556797 18049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:28.566555 18049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:28.567801 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:28.569504 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:28.579748 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:28.581175 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:28.581236 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:28.582127 18049 server_base.cc:1034] running on GCE node
I20260927 05:17:28.583089 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:28.584410 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:28.585671 18049 hybrid_clock.cc:648] HybridClock initialized: now 1790486248585620 us; error 34 us; skew 500 ppm
I20260927 05:17:28.586071 18049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:28.588485 18049 webserver.cc:460] Webserver started at http://127.0.97.62:42659/ using document root <none> and password file <none>
I20260927 05:17:28.589174 18049 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:28.589294 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:28.589635 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:28.592633 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "8a37881a17b04356b30842382f0c2185"
format_stamp: "Formatted at 2026-09-27 05:17:28 on dist-test-slave-1t6p"
server_key: "e725f87bdeb4ea9affac4a3ca7382a4b"
server_key_iv: "21f606ad38e7a45f09ba8684303e4989"
server_key_version: "encryptionkey@0"
I20260927 05:17:28.593431 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a37881a17b04356b30842382f0c2185"
format_stamp: "Formatted at 2026-09-27 05:17:28 on dist-test-slave-1t6p"
server_key: "e725f87bdeb4ea9affac4a3ca7382a4b"
server_key_iv: "21f606ad38e7a45f09ba8684303e4989"
server_key_version: "encryptionkey@0"
I20260927 05:17:28.598440 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:17:28.601795 18063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:28.602993 18049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:17:28.603183 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "8a37881a17b04356b30842382f0c2185"
format_stamp: "Formatted at 2026-09-27 05:17:28 on dist-test-slave-1t6p"
server_key: "e725f87bdeb4ea9affac4a3ca7382a4b"
server_key_iv: "21f606ad38e7a45f09ba8684303e4989"
server_key_version: "encryptionkey@0"
I20260927 05:17:28.603341 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:28.620968 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:28.621639 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:28.621863 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:28.638283 18049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41237
I20260927 05:17:28.638320 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41237 every 8 connection(s)
I20260927 05:17:28.639587 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:28.641462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18049
I20260927 05:17:28.641642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:28.642083 388 external_mini_cluster.cc:1428] Setting key cd0fd251f49ec0b0d58660168d120061
I20260927 05:17:28.643595 18115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:28.655663 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: Bootstrap starting.
I20260927 05:17:28.658664 18115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:28.659869 18115 log.cc:824] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:28.660578 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:28.643973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.97.62:41237)
0927 05:17:28.644875 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:28.644886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:28.644923 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:17:28.646482 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:28.646501 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:28.646919 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:28.647297 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:28.647899 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:28.649142 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:28.651075 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:28.651676 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:28.651806 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:28.658173 (+ 6367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:28.658181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:28.658193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:28.658800 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:28.658914 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:28.658934 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:28.659089 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:28.659298 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 05:17:28.659627 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":343,"threads_started":1}
I20260927 05:17:28.662619 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: No bootstrap required, opened a new log
I20260927 05:17:28.669468 18115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260927 05:17:28.669790 18115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:28.669906 18115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a37881a17b04356b30842382f0c2185, State: Initialized, Role: FOLLOWER
I20260927 05:17:28.670408 18115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260927 05:17:28.670614 18115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:28.670771 18115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:28.670933 18115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:28.672902 18115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260927 05:17:28.673353 18115 leader_election.cc:302] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a37881a17b04356b30842382f0c2185; no voters:
I20260927 05:17:28.674005 18115 leader_election.cc:288] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:28.674144 18120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:28.674916 18120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [term 1 LEADER]: Becoming Leader. State: Replica: 8a37881a17b04356b30842382f0c2185, State: Running, Role: LEADER
I20260927 05:17:28.675385 18120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260927 05:17:28.676048 18115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:28.678213 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a37881a17b04356b30842382f0c2185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260927 05:17:28.678215 18122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a37881a17b04356b30842382f0c2185. Latest consensus state: current_term: 1 leader_uuid: "8a37881a17b04356b30842382f0c2185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a37881a17b04356b30842382f0c2185" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260927 05:17:28.678772 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:28.678817 18122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:28.680845 18127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:28.686131 18127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:28.694147 18127 catalog_manager.cc:1347] Generated new cluster ID: 4b324344e0934f73b8ead431275a9159
I20260927 05:17:28.694310 18127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:28.708557 18127 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:28.709942 18127 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:28.716925 18127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: Generated new TSK 0
I20260927 05:17:28.717629 18127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:28.722898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:29.065136 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:29.065433 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:29.065591 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:29.072635 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:29.072767 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:29.072937 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:29.081881 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:29.083148 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:29.084843 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:29.096637 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.097304 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.097576 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:29.098623 18139 server_base.cc:1034] running on GCE node
I20260927 05:17:29.099187 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:29.100049 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:29.101246 18139 hybrid_clock.cc:648] HybridClock initialized: now 1790486249101190 us; error 60 us; skew 500 ppm
I20260927 05:17:29.101544 18139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:29.103631 18139 webserver.cc:460] Webserver started at http://127.0.97.1:40751/ using document root <none> and password file <none>
I20260927 05:17:29.104272 18139 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:29.104400 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:29.104753 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:29.107668 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "f62e21cfce384be7865a22b3e0a63ce3"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "0fd1967da60e92f172e2fd4cba7e932a"
server_key_iv: "10190a5507366ee400476bf374b170c1"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.108461 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f62e21cfce384be7865a22b3e0a63ce3"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "0fd1967da60e92f172e2fd4cba7e932a"
server_key_iv: "10190a5507366ee400476bf374b170c1"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.113232 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:29.116484 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.117719 18139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 05:17:29.117877 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "f62e21cfce384be7865a22b3e0a63ce3"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "0fd1967da60e92f172e2fd4cba7e932a"
server_key_iv: "10190a5507366ee400476bf374b170c1"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.118080 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:29.134510 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:29.135284 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:29.135551 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:29.136428 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:29.137809 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:29.137921 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.138043 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:29.138137 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.169637 18139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42461
I20260927 05:17:29.169699 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42461 every 8 connection(s)
I20260927 05:17:29.170897 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:29.178510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18139
I20260927 05:17:29.178686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:29.179217 388 external_mini_cluster.cc:1428] Setting key 25fbbc578c24b8db58c8d7669054b900
I20260927 05:17:29.182205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:29.184147 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:29.173507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45037 (local address 127.0.97.62:41237)
0927 05:17:29.173988 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:29.173992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:29.174933 (+ 941us) server_negotiation.cc:407] Connection header received
0927 05:17:29.176984 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:29.176990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:29.177076 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:29.177195 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:29.178182 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.179066 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.181067 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.181675 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.181801 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:29.183183 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:29.183192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:29.183200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:29.183226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:29.183288 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:29.183309 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:29.183407 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:29.183598 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 05:17:29.183725 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":288,"threads_started":1}
I20260927 05:17:29.184739 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:29.173957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.1:45037)
0927 05:17:29.174791 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:29.174828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:29.176695 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:29.177426 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:29.177434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:29.177516 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:29.178025 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:29.178040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.179232 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.179238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:29.180932 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:29.180947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.181889 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.181894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:29.182130 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:29.182851 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:29.182854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:29.182889 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:29.182960 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:29.183408 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:29.183416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:29.183422 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:17:29.183551 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 05:17:29.183802 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":267,"threads_started":1}
I20260927 05:17:29.186159 18266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260927 05:17:29.186439 18266 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:29.187196 18266 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260927 05:17:29.189023 18080 ts_manager.cc:194] Registered new tserver with Master: f62e21cfce384be7865a22b3e0a63ce3 (127.0.97.1:42461)
I20260927 05:17:29.190927 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:45037
W20260927 05:17:29.502416 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:29.502668 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:29.502807 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:29.510115 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:29.510202 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:29.510402 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:29.519244 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:29.520419 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:29.522176 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:29.534757 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.534837 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.534804 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:29.535804 18270 server_base.cc:1034] running on GCE node
I20260927 05:17:29.536335 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:29.537154 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:29.538370 18270 hybrid_clock.cc:648] HybridClock initialized: now 1790486249538331 us; error 51 us; skew 500 ppm
I20260927 05:17:29.538656 18270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:29.540975 18270 webserver.cc:460] Webserver started at http://127.0.97.2:41671/ using document root <none> and password file <none>
I20260927 05:17:29.541571 18270 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:29.541652 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:29.541893 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:29.544498 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "f90999ce88484e628ea0ab8c6dc14a6b"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "a44f1defe6476cd527e08b0b3de3cb42"
server_key_iv: "db17e3e73ba27c3ebdbf47ffe0db47a0"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.545192 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f90999ce88484e628ea0ab8c6dc14a6b"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "a44f1defe6476cd527e08b0b3de3cb42"
server_key_iv: "db17e3e73ba27c3ebdbf47ffe0db47a0"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.549791 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:17:29.553072 18284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.554324 18270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:17:29.554487 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f90999ce88484e628ea0ab8c6dc14a6b"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "a44f1defe6476cd527e08b0b3de3cb42"
server_key_iv: "db17e3e73ba27c3ebdbf47ffe0db47a0"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.554692 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:29.572120 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:29.572815 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:29.573072 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:29.573961 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:29.575380 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:29.575491 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.575613 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:29.575721 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.607746 18270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40377
I20260927 05:17:29.607789 18396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40377 every 8 connection(s)
I20260927 05:17:29.609002 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:29.615365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18270
I20260927 05:17:29.615551 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:29.616178 388 external_mini_cluster.cc:1428] Setting key 8e6537c5cc6d46ff0dcaa12117c9e168
I20260927 05:17:29.618856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:29.622004 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:29.611647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43041 (local address 127.0.97.62:41237)
0927 05:17:29.611828 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:29.611833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:29.613272 (+ 1439us) server_negotiation.cc:407] Connection header received
0927 05:17:29.614977 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:29.614982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:29.615094 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:29.615207 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:29.616418 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.617226 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.619221 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.619618 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.619688 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:29.621187 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:29.621192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:29.621195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:29.621212 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:29.621241 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:29.621259 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:29.621353 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:29.621562 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 05:17:29.621657 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:17:29.622895 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:29.612189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.2:43041)
0927 05:17:29.613105 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:29.613167 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:29.614716 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:29.615432 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:29.615440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:29.615553 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:29.616248 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:29.616288 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.617379 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.617384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:29.618984 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:29.619055 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:29.619917 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:29.619921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:29.620102 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:29.620857 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:29.620860 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:29.620918 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:29.621010 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:29.621351 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:29.621363 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:29.621373 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:17:29.621550 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 05:17:29.621932 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260927 05:17:29.624143 18397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260927 05:17:29.624439 18397 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:29.625273 18397 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260927 05:17:29.626696 18080 ts_manager.cc:194] Registered new tserver with Master: f90999ce88484e628ea0ab8c6dc14a6b (127.0.97.2:40377)
I20260927 05:17:29.627513 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43041
W20260927 05:17:29.939043 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:29.939337 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:29.939498 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:29.946492 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:29.946614 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:29.946792 18400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:29.955595 18400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:29.956768 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:29.958653 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:29.971279 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.971243 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:29.971517 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:29.972460 18400 server_base.cc:1034] running on GCE node
I20260927 05:17:29.973012 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:29.973865 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:29.975103 18400 hybrid_clock.cc:648] HybridClock initialized: now 1790486249975058 us; error 59 us; skew 500 ppm
I20260927 05:17:29.975399 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:29.977559 18400 webserver.cc:460] Webserver started at http://127.0.97.3:38287/ using document root <none> and password file <none>
I20260927 05:17:29.978217 18400 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:29.978350 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:29.978657 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:29.981508 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "22a8627d65a749e4bff20a3a4c90bae6"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "f75ed4fb54039f1e71c9c7ef354744c7"
server_key_iv: "9e47b92471f2f3da8c393787f7fb4505"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.982251 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "22a8627d65a749e4bff20a3a4c90bae6"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "f75ed4fb54039f1e71c9c7ef354744c7"
server_key_iv: "9e47b92471f2f3da8c393787f7fb4505"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.986910 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 05:17:29.990057 18414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:29.991367 18400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:29.991523 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "22a8627d65a749e4bff20a3a4c90bae6"
format_stamp: "Formatted at 2026-09-27 05:17:29 on dist-test-slave-1t6p"
server_key: "f75ed4fb54039f1e71c9c7ef354744c7"
server_key_iv: "9e47b92471f2f3da8c393787f7fb4505"
server_key_version: "encryptionkey@0"
I20260927 05:17:29.991745 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:30.010345 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:30.011234 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:30.011498 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:30.012491 18400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:30.013994 18400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:30.014122 18400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:30.014258 18400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:30.014346 18400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:30.046401 18400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35605
I20260927 05:17:30.046432 18526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35605 every 8 connection(s)
I20260927 05:17:30.047786 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:30.051833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18400
I20260927 05:17:30.052050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:30.052486 388 external_mini_cluster.cc:1428] Setting key dd74fed17e29b5345be3edc51f6d6eed
I20260927 05:17:30.060298 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.050109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52075 (local address 127.0.97.62:41237)
0927 05:17:30.050422 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.050430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.051835 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 05:17:30.054127 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.054132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.054226 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.054336 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:30.055618 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.056375 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.057887 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.058335 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.058482 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.059648 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:30.059653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:30.059656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:30.059672 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:30.059703 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:30.059716 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:30.059785 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.059974 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.060045 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:17:30.061167 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.050763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.3:52075)
0927 05:17:30.051688 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.051728 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.053845 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.054561 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.054572 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.054682 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:30.055465 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.055484 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.056495 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.056499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.057756 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.057766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.058561 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.058566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.058747 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.059381 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:30.059384 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:30.059418 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:30.059488 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:30.059799 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:30.059806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:30.059811 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.059934 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.060187 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
I20260927 05:17:30.062386 18527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260927 05:17:30.062662 18527 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:30.063377 18527 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260927 05:17:30.064630 18080 ts_manager.cc:194] Registered new tserver with Master: 22a8627d65a749e4bff20a3a4c90bae6 (127.0.97.3:35605)
I20260927 05:17:30.065404 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52075
I20260927 05:17:30.067528 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:30.077093 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.071562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.97.62:41237)
0927 05:17:30.071771 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.071775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.072159 (+ 384us) server_negotiation.cc:407] Connection header received
0927 05:17:30.072301 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.072304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.072368 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.072445 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:30.072882 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.073522 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.074956 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.075436 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.075531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.076535 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:30.076540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:30.076544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:30.076559 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:30.076587 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:30.076599 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:30.076684 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.076823 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.076888 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 05:17:30.080673 18080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:30.087383 18080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:30.120956 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.099700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40377 (local address 127.0.0.1:35570)
0927 05:17:30.111643 (+ 11943us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.111663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.111778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.113222 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.113229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.113258 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:30.113645 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.113663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.115662 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.115667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.117718 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.117729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.119920 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.119926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.120164 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.120224 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.120382 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.120579 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11800,"mutex_wait_us":1,"thread_start_us":783,"threads_started":1}
I20260927 05:17:30.121006 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.100333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.97.2:40377)
0927 05:17:30.112719 (+ 12386us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.112725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.112748 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:17:30.112823 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.112828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.113008 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.113209 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:30.113874 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.115504 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.117898 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.119403 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.119523 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.119802 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.120447 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.120624 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12228,"thread_start_us":12118,"threads_started":1}
I20260927 05:17:30.123325 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.101684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35605 (local address 127.0.0.1:35512)
0927 05:17:30.114553 (+ 12869us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.114570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.114698 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.116099 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.116103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.116127 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:30.116651 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.116673 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.118460 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.118465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.120379 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.120399 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.122454 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.122461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.122628 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.122674 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.122778 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.122924 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12757,"mutex_wait_us":125,"thread_start_us":9656,"threads_started":1}
I20260927 05:17:30.123570 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.101970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.97.3:35605)
0927 05:17:30.113932 (+ 11962us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.113940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.114682 (+ 742us) server_negotiation.cc:407] Connection header received
0927 05:17:30.115357 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.115365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.115599 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.115783 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:30.116931 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.118328 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.120607 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.122188 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.122327 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.122668 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.122823 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.122990 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11787,"thread_start_us":11630,"threads_started":1}
I20260927 05:17:30.124009 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.099651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53870)
0927 05:17:30.100458 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.100499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.100662 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.115792 (+ 15130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.115796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.115831 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:30.116206 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.116213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.118125 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.118131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.120335 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.120350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.122926 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.122931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.123113 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.123144 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.123239 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.123390 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":29,"thread_start_us":14651,"threads_started":1}
I20260927 05:17:30.124861 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.099663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.97.1:42461)
0927 05:17:30.115105 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.115111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.115135 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:17:30.115240 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.115250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.115503 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.116359 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:30.116519 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.117986 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.120550 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.122728 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.123675 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.124281 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.124351 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.124517 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15266,"thread_start_us":15164,"threads_started":1}
I20260927 05:17:30.125243 18332 tablet_service.cc:1467] Processing CreateTablet for tablet 2257ab8d82f741d49f3eca24bd312562 (DEFAULT_TABLE table=test-table [id=934504ebfb2d4129bb1acf987b04eaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:30.126724 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257ab8d82f741d49f3eca24bd312562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:30.127230 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 2257ab8d82f741d49f3eca24bd312562 (DEFAULT_TABLE table=test-table [id=934504ebfb2d4129bb1acf987b04eaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:30.127992 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 2257ab8d82f741d49f3eca24bd312562 (DEFAULT_TABLE table=test-table [id=934504ebfb2d4129bb1acf987b04eaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:30.128598 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257ab8d82f741d49f3eca24bd312562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:30.129024 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2257ab8d82f741d49f3eca24bd312562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:30.142419 18545 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Bootstrap starting.
I20260927 05:17:30.143957 18546 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Bootstrap starting.
I20260927 05:17:30.145601 18547 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Bootstrap starting.
I20260927 05:17:30.147414 18546 tablet_bootstrap.cc:654] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:30.147414 18545 tablet_bootstrap.cc:654] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:30.148553 18547 tablet_bootstrap.cc:654] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:30.148680 18546 log.cc:824] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:30.148669 18545 log.cc:824] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:30.149701 18547 log.cc:824] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:30.151723 18545 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: No bootstrap required, opened a new log
I20260927 05:17:30.151714 18546 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: No bootstrap required, opened a new log
I20260927 05:17:30.151979 18546 ts_tablet_manager.cc:1397] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 05:17:30.151979 18545 ts_tablet_manager.cc:1397] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 05:17:30.152230 18547 tablet_bootstrap.cc:492] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: No bootstrap required, opened a new log
I20260927 05:17:30.152487 18547 ts_tablet_manager.cc:1397] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 05:17:30.159790 18547 raft_consensus.cc:348] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.160162 18547 raft_consensus.cc:374] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:30.160260 18547 raft_consensus.cc:729] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a8627d65a749e4bff20a3a4c90bae6, State: Initialized, Role: FOLLOWER
I20260927 05:17:30.160828 18547 consensus_queue.cc:260] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.161871 18527 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260927 05:17:30.162492 18547 ts_tablet_manager.cc:1428] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 05:17:30.162534 18546 raft_consensus.cc:348] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.162550 18545 raft_consensus.cc:348] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.162938 18545 raft_consensus.cc:374] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:30.162937 18546 raft_consensus.cc:374] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:30.163053 18546 raft_consensus.cc:729] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90999ce88484e628ea0ab8c6dc14a6b, State: Initialized, Role: FOLLOWER
I20260927 05:17:30.163056 18545 raft_consensus.cc:729] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62e21cfce384be7865a22b3e0a63ce3, State: Initialized, Role: FOLLOWER
I20260927 05:17:30.163652 18545 consensus_queue.cc:260] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.163652 18546 consensus_queue.cc:260] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.164733 18266 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260927 05:17:30.165378 18545 ts_tablet_manager.cc:1428] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 05:17:30.166293 18397 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260927 05:17:30.166698 18546 ts_tablet_manager.cc:1428] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260927 05:17:30.172928 18267 tablet.cc:2369] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:30.270684 18553 raft_consensus.cc:482] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:30.271088 18553 raft_consensus.cc:504] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.272876 18553 leader_election.cc:288] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f62e21cfce384be7865a22b3e0a63ce3 (127.0.97.1:42461), 22a8627d65a749e4bff20a3a4c90bae6 (127.0.97.3:35605)
I20260927 05:17:30.280292 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.273502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43739 (local address 127.0.97.3:35605)
0927 05:17:30.273780 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.273788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.274220 (+ 432us) server_negotiation.cc:407] Connection header received
0927 05:17:30.274323 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.274330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.274404 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.274491 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:30.275222 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.276267 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.278482 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.279317 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.279443 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.279553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.279928 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.280048 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:17:30.280331 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.273267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35103 (local address 127.0.97.1:42461)
0927 05:17:30.273490 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.273499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.273829 (+ 330us) server_negotiation.cc:407] Connection header received
0927 05:17:30.274144 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.274148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.274252 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.274357 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:30.275112 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.276223 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.278588 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.279433 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.279561 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.279652 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.279964 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.280081 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:17:30.280426 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.273332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35605 (local address 127.0.97.2:43739)
0927 05:17:30.274118 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.274135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.274226 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.274526 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.274533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.274569 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:30.274990 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.275028 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.276389 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.276393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.278278 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.278291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.279531 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.279537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.279754 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.279792 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.279868 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.280005 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":86,"thread_start_us":479,"threads_started":1}
I20260927 05:17:30.280478 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.273008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.97.2:35103)
0927 05:17:30.273694 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.273734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:30.273969 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:30.274380 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:30.274383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:30.274402 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:30.274921 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.274939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.276387 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.276393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.278364 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:30.278400 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.279632 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.279637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:30.279791 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.279813 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:17:30.279920 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:17:30.280041 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":131,"spinlock_wait_cycles":23808,"thread_start_us":494,"threads_started":1}
I20260927 05:17:30.281697 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257ab8d82f741d49f3eca24bd312562" candidate_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62e21cfce384be7865a22b3e0a63ce3" is_pre_election: true
I20260927 05:17:30.281697 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257ab8d82f741d49f3eca24bd312562" candidate_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a8627d65a749e4bff20a3a4c90bae6" is_pre_election: true
I20260927 05:17:30.282261 18221 raft_consensus.cc:2454] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f90999ce88484e628ea0ab8c6dc14a6b in term 0.
I20260927 05:17:30.282261 18482 raft_consensus.cc:2454] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f90999ce88484e628ea0ab8c6dc14a6b in term 0.
I20260927 05:17:30.282979 18287 leader_election.cc:302] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22a8627d65a749e4bff20a3a4c90bae6, f62e21cfce384be7865a22b3e0a63ce3, f90999ce88484e628ea0ab8c6dc14a6b; no voters:
I20260927 05:17:30.283419 18553 raft_consensus.cc:2781] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:30.283563 18553 raft_consensus.cc:482] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:30.283661 18553 raft_consensus.cc:3037] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:30.285766 18553 raft_consensus.cc:504] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.286309 18553 leader_election.cc:288] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [CANDIDATE]: Term 1 election: Requested vote from peers f62e21cfce384be7865a22b3e0a63ce3 (127.0.97.1:42461), 22a8627d65a749e4bff20a3a4c90bae6 (127.0.97.3:35605)
I20260927 05:17:30.286695 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257ab8d82f741d49f3eca24bd312562" candidate_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62e21cfce384be7865a22b3e0a63ce3"
I20260927 05:17:30.286855 18221 raft_consensus.cc:3037] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:30.286849 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2257ab8d82f741d49f3eca24bd312562" candidate_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a8627d65a749e4bff20a3a4c90bae6"
I20260927 05:17:30.287082 18482 raft_consensus.cc:3037] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:30.289007 18221 raft_consensus.cc:2454] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f90999ce88484e628ea0ab8c6dc14a6b in term 1.
I20260927 05:17:30.289312 18482 raft_consensus.cc:2454] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f90999ce88484e628ea0ab8c6dc14a6b in term 1.
I20260927 05:17:30.289377 18288 leader_election.cc:302] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f62e21cfce384be7865a22b3e0a63ce3, f90999ce88484e628ea0ab8c6dc14a6b; no voters:
I20260927 05:17:30.289671 18553 raft_consensus.cc:2781] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:30.290087 18553 raft_consensus.cc:686] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 1 LEADER]: Becoming Leader. State: Replica: f90999ce88484e628ea0ab8c6dc14a6b, State: Running, Role: LEADER
I20260927 05:17:30.290584 18553 consensus_queue.cc:237] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } }
I20260927 05:17:30.295279 18080 catalog_manager.cc:5495] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b reported cstate change: term changed from 0 to 1, leader changed from <none> to f90999ce88484e628ea0ab8c6dc14a6b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f90999ce88484e628ea0ab8c6dc14a6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 } health_report { overall_health: UNKNOWN } } }
W20260927 05:17:30.299345 18528 tablet.cc:2369] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:30.352557 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.347223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.0.97.2:40377)
0927 05:17:30.347446 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.347452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.347561 (+ 109us) server_negotiation.cc:407] Connection header received
0927 05:17:30.347840 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.347844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.347927 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.348073 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:30.348441 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.349117 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.350468 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.350869 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.351000 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.351527 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:30.351990 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:30.352146 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.352211 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.352283 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:17:30.360673 18482 raft_consensus.cc:1261] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 1 FOLLOWER]: Refusing update from remote peer f90999ce88484e628ea0ab8c6dc14a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:30.360978 18221 raft_consensus.cc:1261] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 1 FOLLOWER]: Refusing update from remote peer f90999ce88484e628ea0ab8c6dc14a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 05:17:30.360970 18398 tablet.cc:2369] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:30.361716 18556 consensus_queue.cc:1035] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a8627d65a749e4bff20a3a4c90bae6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:30.362054 18553 consensus_queue.cc:1035] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62e21cfce384be7865a22b3e0a63ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:30.373163 18559 mvcc.cc:205] Tried to move back new op lower bound from 7333831681466667008 to 7333831681194680320. Current Snapshot: MvccSnapshot[applied={T|T < 7333831681466667008}]
I20260927 05:17:30.386883 18079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:30.387233 18079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:30.390863 18079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a37881a17b04356b30842382f0c2185: Sending DeleteTablet for 3 replicas of tablet 2257ab8d82f741d49f3eca24bd312562
I20260927 05:17:30.391928 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257ab8d82f741d49f3eca24bd312562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:30 UTC) from {username='slave'} at 127.0.0.1:53870
I20260927 05:17:30.392025 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257ab8d82f741d49f3eca24bd312562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:30 UTC) from {username='slave'} at 127.0.0.1:35570
I20260927 05:17:30.392313 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 2257ab8d82f741d49f3eca24bd312562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:30 UTC) from {username='slave'} at 127.0.0.1:35512
I20260927 05:17:30.394260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18139
I20260927 05:17:30.428079 18569 tablet_replica.cc:330] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3: stopping tablet replica
I20260927 05:17:30.428721 18571 tablet_replica.cc:330] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: stopping tablet replica
I20260927 05:17:30.429324 18570 tablet_replica.cc:330] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: stopping tablet replica
I20260927 05:17:30.429769 18570 raft_consensus.cc:2229] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:30.430488 18570 raft_consensus.cc:2258] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:30.430867 18571 raft_consensus.cc:2229] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:30.430684 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.396138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.0.97.1:42461)
0927 05:17:30.396345 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.396354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.418957 (+ 22603us) server_negotiation.cc:407] Connection header received
0927 05:17:30.419528 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.419537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.419661 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.419744 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:30.420615 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.422139 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.425417 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.426338 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.426461 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.428974 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:30.428987 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:30.429007 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:30.429770 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:30.429836 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:30.429864 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:30.429970 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.430203 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.430314 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:30.431263 18569 raft_consensus.cc:2229] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:30.431792 18569 raft_consensus.cc:2258] T 2257ab8d82f741d49f3eca24bd312562 P f62e21cfce384be7865a22b3e0a63ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:30.431809 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:30.433065 18570 ts_tablet_manager.cc:1905] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:30.435384 18571 raft_consensus.cc:2258] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:30.437427 18571 ts_tablet_manager.cc:1905] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:30.439487 18570 ts_tablet_manager.cc:1918] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:30.439677 18570 log.cc:1196] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/2257ab8d82f741d49f3eca24bd312562
I20260927 05:17:30.440291 18570 ts_tablet_manager.cc:1939] T 2257ab8d82f741d49f3eca24bd312562 P f90999ce88484e628ea0ab8c6dc14a6b: Deleting consensus metadata
I20260927 05:17:30.442000 18065 catalog_manager.cc:4841] TS f90999ce88484e628ea0ab8c6dc14a6b (127.0.97.2:40377): tablet 2257ab8d82f741d49f3eca24bd312562 (table test-table [id=934504ebfb2d4129bb1acf987b04eaf3]) successfully deleted
I20260927 05:17:30.445487 18571 ts_tablet_manager.cc:1918] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:30.445719 18571 log.cc:1196] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/2257ab8d82f741d49f3eca24bd312562
I20260927 05:17:30.446218 18571 ts_tablet_manager.cc:1939] T 2257ab8d82f741d49f3eca24bd312562 P 22a8627d65a749e4bff20a3a4c90bae6: Deleting consensus metadata
I20260927 05:17:30.447793 18066 catalog_manager.cc:4841] TS 22a8627d65a749e4bff20a3a4c90bae6 (127.0.97.3:35605): tablet 2257ab8d82f741d49f3eca24bd312562 (table test-table [id=934504ebfb2d4129bb1acf987b04eaf3]) successfully deleted
I20260927 05:17:30.531467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18139
W20260927 05:17:30.548868 18067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42461 (error 108)
I20260927 05:17:30.550181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18270
I20260927 05:17:30.551065 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.550543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53894)
0927 05:17:30.550734 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.550859 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:17:30.551384 18067 catalog_manager.cc:4584] TS f62e21cfce384be7865a22b3e0a63ce3 (127.0.97.1:42461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2257ab8d82f741d49f3eca24bd312562: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
I20260927 05:17:30.557384 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.550897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.97.2:40377)
0927 05:17:30.551093 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.551100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.551366 (+ 266us) server_negotiation.cc:407] Connection header received
0927 05:17:30.551426 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.551429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.551503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.551618 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:30.552033 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.552861 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.554185 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.554615 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.554760 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.556048 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:30.556057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:30.556070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:30.556643 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:30.556688 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:30.556707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:30.556843 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.556944 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.557033 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:17:30.558065 18392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:30.604933 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.604395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53906)
0927 05:17:30.604635 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.604740 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:17:30.606070 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.605677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53914)
0927 05:17:30.605840 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.605924 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:17:30.666574 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.666072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53920)
0927 05:17:30.666297 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.666402 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:17:30.667663 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.667271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53928)
0927 05:17:30.667428 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.667523 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:17:30.670773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18270
I20260927 05:17:30.689980 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18400
I20260927 05:17:30.696859 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.690725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.97.3:35605)
0927 05:17:30.690903 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:30.690911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:30.690991 (+ 80us) server_negotiation.cc:407] Connection header received
0927 05:17:30.691232 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:30.691236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:30.691320 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:30.691425 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:30.691826 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.692518 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.693793 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:30.694174 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:30.694299 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:30.695487 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:30.695495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:30.695509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:30.696140 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:30.696186 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:30.696206 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:30.696328 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:30.696454 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 05:17:30.696556 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:30.697482 18522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:30.748386 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.747838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53930)
0927 05:17:30.748085 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.748195 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 05:17:30.749529 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:30.749087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42461 (local address 127.0.0.1:53934)
0927 05:17:30.749307 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:30.749396 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 05:17:30.784462 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18400
I20260927 05:17:30.800513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18049
I20260927 05:17:30.801563 18110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:30.908555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18049
2026-09-27T05:17:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2747 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:31 dist-test-slave-1t6p krb5kdc[18580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:31 dist-test-slave-1t6p krb5kdc[18580](info): set up 2 sockets
Sep 27 05:17:31 dist-test-slave-1t6p krb5kdc[18580](info): commencing operation
krb5kdc: starting...
W20260927 05:17:32.030958 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:32 dist-test-slave-1t6p krb5kdc[18580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:17:32.258402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46565
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:32.570629 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:32.570883 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:32.570966 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:32.577754 18593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:32.577845 18593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:32.577891 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:32.577946 18593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:32.577991 18593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:32.587289 18593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:32.588366 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:32.589550 18593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:32.600766 18598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:32.600699 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:32.600740 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:32.601665 18593 server_base.cc:1034] running on GCE node
I20260927 05:17:32.602316 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:32.603660 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:32.604861 18593 hybrid_clock.cc:648] HybridClock initialized: now 1790486252604804 us; error 50 us; skew 500 ppm
I20260927 05:17:32.605199 18593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:32 dist-test-slave-1t6p krb5kdc[18580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:32.608186 18593 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:17:32.609470 18593 webserver.cc:460] Webserver started at http://127.0.97.62:46367/ using document root <none> and password file <none>
I20260927 05:17:32.610054 18593 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:32.610145 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:32.610394 18593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:32.613139 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be"
format_stamp: "Formatted at 2026-09-27 05:17:32 on dist-test-slave-1t6p"
server_key: "49b4a6e0360e9d2c2c297ca739a34344"
server_key_iv: "340b15c7d9a97884a8a6dfcc37d650ff"
server_key_version: "encryptionkey@0"
I20260927 05:17:32.613801 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be"
format_stamp: "Formatted at 2026-09-27 05:17:32 on dist-test-slave-1t6p"
server_key: "49b4a6e0360e9d2c2c297ca739a34344"
server_key_iv: "340b15c7d9a97884a8a6dfcc37d650ff"
server_key_version: "encryptionkey@0"
I20260927 05:17:32.618423 18593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:17:32.621649 18608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:32.622889 18593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:17:32.623073 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be"
format_stamp: "Formatted at 2026-09-27 05:17:32 on dist-test-slave-1t6p"
server_key: "49b4a6e0360e9d2c2c297ca739a34344"
server_key_iv: "340b15c7d9a97884a8a6dfcc37d650ff"
server_key_version: "encryptionkey@0"
I20260927 05:17:32.623293 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:32.645408 18593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:32.649276 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:32.649523 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:32.667328 18593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46565
I20260927 05:17:32.667368 18659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46565 every 8 connection(s)
I20260927 05:17:32.668685 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:32.672180 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:32.678133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18593
I20260927 05:17:32.678339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:32.678803 388 external_mini_cluster.cc:1428] Setting key 639e8cca1c24b7060603568d1389696e
I20260927 05:17:32.682266 18660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: Bootstrap starting.
I20260927 05:17:32.685678 18660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:32.686726 18660 log.cc:824] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:32.689244 18660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: No bootstrap required, opened a new log
Sep 27 05:17:32 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:32.696774 18660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } }
I20260927 05:17:32.697113 18660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:32.697213 18660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4660a5244ddd4e55a0ca2ffc8e65d7be, State: Initialized, Role: FOLLOWER
I20260927 05:17:32.697786 18660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } }
I20260927 05:17:32.698028 18660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:32.698187 18660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:32.698374 18660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:32.701267 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:32.680778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.0.97.62:46565)
0927 05:17:32.681519 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:32.681530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:32.681568 (+ 38us) server_negotiation.cc:407] Connection header received
0927 05:17:32.683361 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:32.683383 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:32.683743 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:32.684041 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:32.685886 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:32.687297 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:32.688696 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:32.689388 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:32.689566 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:32.693543 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:32.693554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:32.693568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:32.693602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:32.696622 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:32.697466 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:32.697472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:32.697478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:32.697702 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:32.698096 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:32.698103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:32.698108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:32.698817 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:32.699142 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:32.699586 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 05:17:32.700023 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
I20260927 05:17:32.701532 18660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } }
I20260927 05:17:32.702033 18660 leader_election.cc:302] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4660a5244ddd4e55a0ca2ffc8e65d7be; no voters:
I20260927 05:17:32.702656 18660 leader_election.cc:288] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:32.702726 18665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:32.704095 18665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [term 1 LEADER]: Becoming Leader. State: Replica: 4660a5244ddd4e55a0ca2ffc8e65d7be, State: Running, Role: LEADER
I20260927 05:17:32.704578 18665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } }
I20260927 05:17:32.704749 18660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:32.707934 18667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4660a5244ddd4e55a0ca2ffc8e65d7be. Latest consensus state: current_term: 1 leader_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } } }
I20260927 05:17:32.707782 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4660a5244ddd4e55a0ca2ffc8e65d7be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46565 } } }
I20260927 05:17:32.708424 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:32.708420 18667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:32.710040 18674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:32.715072 18674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:32.722728 18674 catalog_manager.cc:1347] Generated new cluster ID: b9782843c569495fa709212f139e42cf
I20260927 05:17:32.722834 18674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:32.737169 18674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:32.738547 18674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:32.760871 18674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: Generated new TSK 0
I20260927 05:17:32.761668 18674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:17:32.823125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:33.144704 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:33.144953 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:33.145090 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:33.151980 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:33.152070 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:33.152287 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:33.161377 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:33.162529 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:33.163803 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:33.176360 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:33.176393 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:33.176424 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:33.177748 18686 server_base.cc:1034] running on GCE node
I20260927 05:17:33.178380 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:33.179323 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:33.180517 18686 hybrid_clock.cc:648] HybridClock initialized: now 1790486253180488 us; error 29 us; skew 500 ppm
I20260927 05:17:33.180811 18686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:33 dist-test-slave-1t6p krb5kdc[18580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:33.183809 18686 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:17:33.185168 18686 webserver.cc:460] Webserver started at http://127.0.97.1:39351/ using document root <none> and password file <none>
I20260927 05:17:33.185824 18686 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:33.185950 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:33.186249 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:33.189085 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "790b37f30c034dbdb6490e3dccc043c3"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "73740ad03d77d67b29d7dbe206622c0c"
server_key_iv: "9d358de0aaa30bafb45b3c122c220897"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.189836 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "790b37f30c034dbdb6490e3dccc043c3"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "73740ad03d77d67b29d7dbe206622c0c"
server_key_iv: "9d358de0aaa30bafb45b3c122c220897"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.194550 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:17:33.197921 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.199189 18686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:33.199349 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "790b37f30c034dbdb6490e3dccc043c3"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "73740ad03d77d67b29d7dbe206622c0c"
server_key_iv: "9d358de0aaa30bafb45b3c122c220897"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.199522 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:33.214649 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:33.218465 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:33.218742 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:33.219686 18686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:33.221139 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:33.221256 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.221395 18686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:33.221457 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.254278 18686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40487
I20260927 05:17:33.254320 18813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40487 every 8 connection(s)
I20260927 05:17:33.255563 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:33.264381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18686
I20260927 05:17:33.264568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:33.265087 388 external_mini_cluster.cc:1428] Setting key 595e20fa175dfc5103fdf1c82c480626
Sep 27 05:17:33 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:33.276111 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:33.257954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37319 (local address 127.0.97.62:46565)
0927 05:17:33.258389 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:33.258394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:33.259279 (+ 885us) server_negotiation.cc:407] Connection header received
0927 05:17:33.261052 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:33.261058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:33.261147 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:33.261271 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:33.263034 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.263804 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.266021 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.266611 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.266733 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:33.270608 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:33.270613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:33.270616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:33.270647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:33.273135 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:33.273843 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:33.273855 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:33.273858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:33.273972 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:33.274380 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:33.274386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:33.274389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:33.274637 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:33.274793 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:33.275530 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 05:17:33.275742 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260927 05:17:33.276973 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:33.258378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46565 (local address 127.0.97.1:37319)
0927 05:17:33.259117 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:33.259158 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:33.260784 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:33.261497 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:33.261505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:33.261984 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:33.262781 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:33.262808 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.263944 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.263947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:33.265872 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:33.265886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.266852 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.266863 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:33.267161 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:33.267827 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:33.267861 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:33.270406 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:33.273294 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:33.273302 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:33.273317 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:33.273718 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:33.274080 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:33.274085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:33.274089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:33.274269 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:33.274770 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:33.274777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:33.275157 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 05:17:33.275433 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 05:17:33.275933 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260927 05:17:33.278390 18814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46565
I20260927 05:17:33.278702 18814 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:33.279539 18814 heartbeater.cc:511] Master 127.0.97.62:46565 requested a full tablet report, sending...
I20260927 05:17:33.281598 18625 ts_manager.cc:194] Registered new tserver with Master: 790b37f30c034dbdb6490e3dccc043c3 (127.0.97.1:40487)
I20260927 05:17:33.283104 18625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37319
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:17:33.318518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:33.635075 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:33.635377 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:33.635546 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:33.642475 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:33.642607 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:33.642793 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:33.651878 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:33.653076 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:33.654361 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:33.668258 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:33.668880 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:33.669185 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:33.674389 18820 server_base.cc:1034] running on GCE node
I20260927 05:17:33.675067 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:33.676021 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:33.677232 18820 hybrid_clock.cc:648] HybridClock initialized: now 1790486253677175 us; error 51 us; skew 500 ppm
I20260927 05:17:33.677551 18820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:33 dist-test-slave-1t6p krb5kdc[18580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:33.683861 18820 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:17:33.685353 18820 webserver.cc:460] Webserver started at http://127.0.97.2:39091/ using document root <none> and password file <none>
I20260927 05:17:33.686080 18820 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:33.686223 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:33.686522 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:33.689316 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "991966b2845f4247a02deaf595ad2706"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "66955c813642b04314c0564095ea0a42"
server_key_iv: "2248a291797355b158462bcd8cd0af52"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.690080 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "991966b2845f4247a02deaf595ad2706"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "66955c813642b04314c0564095ea0a42"
server_key_iv: "2248a291797355b158462bcd8cd0af52"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.694980 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:17:33.698360 18835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.699684 18820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:33.699859 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "991966b2845f4247a02deaf595ad2706"
format_stamp: "Formatted at 2026-09-27 05:17:33 on dist-test-slave-1t6p"
server_key: "66955c813642b04314c0564095ea0a42"
server_key_iv: "2248a291797355b158462bcd8cd0af52"
server_key_version: "encryptionkey@0"
I20260927 05:17:33.700057 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:33.720096 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:33.724025 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:33.724303 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:33.725203 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:33.726517 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:33.726620 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.726786 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:33.726866 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:33.759277 18820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42891
I20260927 05:17:33.759326 18947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42891 every 8 connection(s)
I20260927 05:17:33.760478 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:33.769004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18820
I20260927 05:17:33.769201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:33.769629 388 external_mini_cluster.cc:1428] Setting key 4cbf76ab1c689a693eea7c6abfc02068
Sep 27 05:17:33 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:33.780598 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:33.762931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48841 (local address 127.0.97.62:46565)
0927 05:17:33.763160 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:33.763165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:33.764250 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 05:17:33.766049 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:33.766055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:33.766143 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:33.766251 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:33.767689 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.768385 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.770055 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.770516 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.770662 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:33.774593 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:33.774598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:33.774601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:33.774630 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:33.777115 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:33.778037 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:33.778041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:33.778043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:33.778161 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:33.778684 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:33.778687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:33.778689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:33.778928 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:33.779098 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:33.779924 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 05:17:33.780194 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 05:17:33.781236 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:33.763443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46565 (local address 127.0.97.2:48841)
0927 05:17:33.764094 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:33.764137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:33.765798 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:33.766473 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:33.766481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:33.766959 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:33.767531 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:33.767547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.768547 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.768552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:33.769927 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:33.769939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:33.770744 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:33.770749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:33.770940 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:33.771707 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:33.771739 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:33.774375 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:33.777259 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:33.777279 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:33.777296 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:33.777909 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:33.778308 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:33.778316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:33.778322 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:33.778570 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:33.779070 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:33.779080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:33.779509 (+ 429us) client_negotiation.cc:769] Sending connection context
0927 05:17:33.779805 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 05:17:33.780220 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":257,"threads_started":1}
I20260927 05:17:33.782615 18948 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46565
I20260927 05:17:33.782910 18948 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:33.783562 18948 heartbeater.cc:511] Master 127.0.97.62:46565 requested a full tablet report, sending...
I20260927 05:17:33.784937 18625 ts_manager.cc:194] Registered new tserver with Master: 991966b2845f4247a02deaf595ad2706 (127.0.97.2:42891)
I20260927 05:17:33.786157 18625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48841
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:17:33.821553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:34.138289 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:34.138566 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:34.138662 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:34.145684 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:34.145783 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:34.145892 18953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:34.154987 18953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:34.156116 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:34.157370 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:34.169790 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:34.169872 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:34.169852 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:34.170804 18953 server_base.cc:1034] running on GCE node
I20260927 05:17:34.171471 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:34.172384 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:34.173626 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790486254173553 us; error 82 us; skew 500 ppm
I20260927 05:17:34.173940 18953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:34 dist-test-slave-1t6p krb5kdc[18580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:34.177042 18953 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:17:34.178483 18953 webserver.cc:460] Webserver started at http://127.0.97.3:34841/ using document root <none> and password file <none>
I20260927 05:17:34.179214 18953 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:34.179343 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:34.179672 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:34.182374 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "8eff2436753f43e8b5cca0a99ecb5a10"
format_stamp: "Formatted at 2026-09-27 05:17:34 on dist-test-slave-1t6p"
server_key: "5be9abd6fc558de916abee44f5ca225d"
server_key_iv: "468c2adead0cc90aa559c52e4a031037"
server_key_version: "encryptionkey@0"
I20260927 05:17:34.183200 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8eff2436753f43e8b5cca0a99ecb5a10"
format_stamp: "Formatted at 2026-09-27 05:17:34 on dist-test-slave-1t6p"
server_key: "5be9abd6fc558de916abee44f5ca225d"
server_key_iv: "468c2adead0cc90aa559c52e4a031037"
server_key_version: "encryptionkey@0"
I20260927 05:17:34.187978 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:17:34.191329 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:34.192538 18953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:17:34.192742 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "8eff2436753f43e8b5cca0a99ecb5a10"
format_stamp: "Formatted at 2026-09-27 05:17:34 on dist-test-slave-1t6p"
server_key: "5be9abd6fc558de916abee44f5ca225d"
server_key_iv: "468c2adead0cc90aa559c52e4a031037"
server_key_version: "encryptionkey@0"
I20260927 05:17:34.192931 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:34.204133 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:34.208316 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:34.208593 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:34.209468 18953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:34.210793 18953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:34.210891 18953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:34.211048 18953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:34.211133 18953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 05:17:34.243927 18953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34891
I20260927 05:17:34.243968 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34891 every 8 connection(s)
I20260927 05:17:34.245172 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:34.250979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 18953
I20260927 05:17:34.251235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:34.251670 388 external_mini_cluster.cc:1428] Setting key 71c381fcd67fa7c33c81c46edfe00877
Sep 27 05:17:34 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:34.264863 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.247655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56577 (local address 127.0.97.62:46565)
0927 05:17:34.247833 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.247839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.248964 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 05:17:34.250829 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.250835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.250922 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.251050 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:34.252733 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.253828 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.255632 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.256056 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.256217 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.259782 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:34.259787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:34.259790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:34.259823 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:34.262192 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.262859 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.262864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.262866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.262986 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.263426 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.263429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.263431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.263682 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:34.263802 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.264380 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.264540 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:34.265723 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.248137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46565 (local address 127.0.97.3:56577)
0927 05:17:34.248831 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.248871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.250558 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.251275 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.251283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.251847 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:34.252542 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.252565 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.253985 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.253991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.255482 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.255498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.256291 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.256301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.256545 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.257239 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:34.257261 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:34.259567 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:34.262325 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:34.262331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:34.262341 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:34.262747 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:34.263142 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:34.263146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:34.263148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:34.263326 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:34.263794 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:34.263799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:34.264067 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.264285 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.264619 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":265,"threads_started":1}
I20260927 05:17:34.267190 19081 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46565
I20260927 05:17:34.267501 19081 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:34.268138 19081 heartbeater.cc:511] Master 127.0.97.62:46565 requested a full tablet report, sending...
I20260927 05:17:34.269480 18625 ts_manager.cc:194] Registered new tserver with Master: 8eff2436753f43e8b5cca0a99ecb5a10 (127.0.97.3:34891)
I20260927 05:17:34.270251 18625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56577
I20260927 05:17:34.278913 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:34.286514 18814 heartbeater.cc:503] Master 127.0.97.62:46565 was elected leader, sending a full tablet report...
I20260927 05:17:34.295513 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.282901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.97.62:46565)
0927 05:17:34.283086 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.283105 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.283540 (+ 435us) server_negotiation.cc:407] Connection header received
0927 05:17:34.283725 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.283730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.283799 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.283895 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:34.285202 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.285916 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.287622 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.288158 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.288252 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.290653 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:34.290658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:34.290661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:34.290691 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:34.293028 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.293720 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.293724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.293726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.293832 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.294200 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.294203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.294205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.294604 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:34.294802 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.295172 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.295297 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:17:34.299230 18625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:34.306104 18625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:34.335322 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.320044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.97.3:34891)
0927 05:17:34.326708 (+ 6664us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.326716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.327404 (+ 688us) server_negotiation.cc:407] Connection header received
0927 05:17:34.327614 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.327621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.327841 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.328038 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:34.328726 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.330260 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.332350 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.333800 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.333921 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.334157 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.334754 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.334909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6502,"thread_start_us":6343,"threads_started":1}
I20260927 05:17:34.335399 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.318991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49266)
0927 05:17:34.326280 (+ 7289us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.326337 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.326461 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.327571 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.327578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.327608 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:34.327999 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.328012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.329771 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.329774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.332467 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.332481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.334454 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.334459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.334587 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.334610 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.334680 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.334915 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7144,"mutex_wait_us":225,"thread_start_us":7033,"threads_started":1}
I20260927 05:17:34.335940 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.317950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42891 (local address 127.0.0.1:53132)
0927 05:17:34.325002 (+ 7052us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.325046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.325217 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.328481 (+ 3264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.328485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.328510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:34.328869 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.328880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.330953 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.330961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.333022 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.333032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.334883 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.334888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.335069 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.335102 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.335184 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.335320 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6833,"mutex_wait_us":3,"thread_start_us":6694,"threads_started":1}
I20260927 05:17:34.335938 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.318861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.97.1:40487)
0927 05:17:34.325844 (+ 6983us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.325856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.326583 (+ 727us) server_negotiation.cc:407] Connection header received
0927 05:17:34.326702 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.326710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.326991 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.327200 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:34.328233 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.329593 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.332617 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.333991 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.334096 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.334320 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.335602 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.335723 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6736,"thread_start_us":6537,"threads_started":1}
I20260927 05:17:34.336340 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.319392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34891 (local address 127.0.0.1:46486)
0927 05:17:34.327156 (+ 7764us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.327172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.327273 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.328046 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.328051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.328075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:34.328490 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.328503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.330428 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.330434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.332193 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.332205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.334276 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.334285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.334506 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.334555 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.334680 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.335197 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7618,"mutex_wait_us":322,"thread_start_us":7463,"threads_started":1}
I20260927 05:17:34.337481 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.317972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.97.2:42891)
0927 05:17:34.327898 (+ 9926us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.327904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.327924 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:17:34.328038 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.328044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.328251 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.328478 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:34.329032 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.330818 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.333174 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.334628 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.334772 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.334936 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.337020 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.337193 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9769,"thread_start_us":9668,"threads_started":1}
I20260927 05:17:34.339699 19016 tablet_service.cc:1467] Processing CreateTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 (DEFAULT_TABLE table=test-table [id=9ec3d9858dcb40a4b05190a702964894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:34.339748 18749 tablet_service.cc:1467] Processing CreateTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 (DEFAULT_TABLE table=test-table [id=9ec3d9858dcb40a4b05190a702964894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:34.340842 19016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52afaab1b749ea95c3d90cf3812c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:34.340499 18883 tablet_service.cc:1467] Processing CreateTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 (DEFAULT_TABLE table=test-table [id=9ec3d9858dcb40a4b05190a702964894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:34.341091 18749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52afaab1b749ea95c3d90cf3812c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:34.341670 18883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52afaab1b749ea95c3d90cf3812c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:34.351903 19099 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Bootstrap starting.
I20260927 05:17:34.355087 19100 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Bootstrap starting.
I20260927 05:17:34.355151 19099 tablet_bootstrap.cc:654] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:34.356874 19101 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Bootstrap starting.
I20260927 05:17:34.356930 19099 log.cc:824] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:34.359617 19100 tablet_bootstrap.cc:654] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:34.359920 19101 tablet_bootstrap.cc:654] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:34.360226 19099 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: No bootstrap required, opened a new log
I20260927 05:17:34.360491 19099 ts_tablet_manager.cc:1397] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:17:34.360668 19100 log.cc:824] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:34.361048 19101 log.cc:824] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:34.363472 19101 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: No bootstrap required, opened a new log
I20260927 05:17:34.363624 19100 tablet_bootstrap.cc:492] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: No bootstrap required, opened a new log
I20260927 05:17:34.363765 19101 ts_tablet_manager.cc:1397] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 05:17:34.363881 19100 ts_tablet_manager.cc:1397] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 05:17:34.370827 19101 raft_consensus.cc:348] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.370900 19099 raft_consensus.cc:348] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.371250 19101 raft_consensus.cc:374] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:34.371361 19099 raft_consensus.cc:374] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:34.371361 19101 raft_consensus.cc:729] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991966b2845f4247a02deaf595ad2706, State: Initialized, Role: FOLLOWER
I20260927 05:17:34.371450 19099 raft_consensus.cc:729] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eff2436753f43e8b5cca0a99ecb5a10, State: Initialized, Role: FOLLOWER
I20260927 05:17:34.371924 19101 consensus_queue.cc:260] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.372051 19099 consensus_queue.cc:260] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.372997 18948 heartbeater.cc:503] Master 127.0.97.62:46565 was elected leader, sending a full tablet report...
I20260927 05:17:34.373621 19101 ts_tablet_manager.cc:1428] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 05:17:34.375097 19100 raft_consensus.cc:348] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.375595 19100 raft_consensus.cc:374] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:34.375715 19100 raft_consensus.cc:729] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790b37f30c034dbdb6490e3dccc043c3, State: Initialized, Role: FOLLOWER
I20260927 05:17:34.376381 19100 consensus_queue.cc:260] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.378393 19100 ts_tablet_manager.cc:1428] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 05:17:34.379109 19081 heartbeater.cc:503] Master 127.0.97.62:46565 was elected leader, sending a full tablet report...
I20260927 05:17:34.379971 19099 ts_tablet_manager.cc:1428] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260927 05:17:34.496855 19082 tablet.cc:2369] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:34.507876 18815 tablet.cc:2369] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:34.512329 18949 tablet.cc:2369] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:34.675410 19107 raft_consensus.cc:482] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:34.675876 19107 raft_consensus.cc:504] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.677956 19107 leader_election.cc:288] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 991966b2845f4247a02deaf595ad2706 (127.0.97.2:42891), 8eff2436753f43e8b5cca0a99ecb5a10 (127.0.97.3:34891)
I20260927 05:17:34.685274 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.678961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46849 (local address 127.0.97.2:42891)
0927 05:17:34.679176 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.679195 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.679670 (+ 475us) server_negotiation.cc:407] Connection header received
0927 05:17:34.679993 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.680001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.680094 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.680187 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:34.680730 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.681826 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.683636 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.684363 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.684511 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.684603 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.684883 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.685000 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:17:34.685395 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.678509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60045 (local address 127.0.97.3:34891)
0927 05:17:34.678703 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.678709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.680122 (+ 1413us) server_negotiation.cc:407] Connection header received
0927 05:17:34.680311 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.680318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.680387 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.680436 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:34.681254 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.681989 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.683755 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.684498 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.684647 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.684757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.685012 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.685131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:17:34.685421 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.678152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42891 (local address 127.0.97.1:46849)
0927 05:17:34.679551 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.679573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.679802 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.680213 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.680219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.680245 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:34.680609 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.680619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.681949 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.681956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.683484 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.683496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.684562 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.684567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.684724 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.684753 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.684857 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.684975 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":7,"thread_start_us":1509,"threads_started":1}
I20260927 05:17:34.685478 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.678348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34891 (local address 127.0.97.1:60045)
0927 05:17:34.679994 (+ 1646us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:34.680010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:34.680129 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:34.680675 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:34.680681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:34.680705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:34.681061 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.681072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.682139 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.682145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.683603 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:34.683614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.684713 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.684719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:34.684862 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.684882 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 05:17:34.684988 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 05:17:34.685108 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":884,"threads_started":1}
I20260927 05:17:34.686316 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52afaab1b749ea95c3d90cf3812c64" candidate_uuid: "790b37f30c034dbdb6490e3dccc043c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991966b2845f4247a02deaf595ad2706" is_pre_election: true
I20260927 05:17:34.686429 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52afaab1b749ea95c3d90cf3812c64" candidate_uuid: "790b37f30c034dbdb6490e3dccc043c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" is_pre_election: true
I20260927 05:17:34.686833 18903 raft_consensus.cc:2454] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790b37f30c034dbdb6490e3dccc043c3 in term 0.
I20260927 05:17:34.686991 19036 raft_consensus.cc:2454] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790b37f30c034dbdb6490e3dccc043c3 in term 0.
I20260927 05:17:34.687640 18703 leader_election.cc:302] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 790b37f30c034dbdb6490e3dccc043c3, 8eff2436753f43e8b5cca0a99ecb5a10; no voters:
I20260927 05:17:34.688133 19107 raft_consensus.cc:2781] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:34.688282 19107 raft_consensus.cc:482] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:34.688367 19107 raft_consensus.cc:3037] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:34.690418 19107 raft_consensus.cc:504] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.691028 19107 leader_election.cc:288] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [CANDIDATE]: Term 1 election: Requested vote from peers 991966b2845f4247a02deaf595ad2706 (127.0.97.2:42891), 8eff2436753f43e8b5cca0a99ecb5a10 (127.0.97.3:34891)
I20260927 05:17:34.691417 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52afaab1b749ea95c3d90cf3812c64" candidate_uuid: "790b37f30c034dbdb6490e3dccc043c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991966b2845f4247a02deaf595ad2706"
I20260927 05:17:34.691514 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52afaab1b749ea95c3d90cf3812c64" candidate_uuid: "790b37f30c034dbdb6490e3dccc043c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eff2436753f43e8b5cca0a99ecb5a10"
I20260927 05:17:34.691627 18903 raft_consensus.cc:3037] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:34.691701 19036 raft_consensus.cc:3037] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:34.694887 18903 raft_consensus.cc:2454] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790b37f30c034dbdb6490e3dccc043c3 in term 1.
I20260927 05:17:34.694887 19036 raft_consensus.cc:2454] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790b37f30c034dbdb6490e3dccc043c3 in term 1.
I20260927 05:17:34.695297 18703 leader_election.cc:302] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 790b37f30c034dbdb6490e3dccc043c3, 8eff2436753f43e8b5cca0a99ecb5a10, 991966b2845f4247a02deaf595ad2706; no voters:
I20260927 05:17:34.695684 19107 raft_consensus.cc:2781] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:34.696110 19107 raft_consensus.cc:686] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 1 LEADER]: Becoming Leader. State: Replica: 790b37f30c034dbdb6490e3dccc043c3, State: Running, Role: LEADER
I20260927 05:17:34.696640 19107 consensus_queue.cc:237] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } }
I20260927 05:17:34.700905 18625 catalog_manager.cc:5495] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 790b37f30c034dbdb6490e3dccc043c3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "790b37f30c034dbdb6490e3dccc043c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790b37f30c034dbdb6490e3dccc043c3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 } health_report { overall_health: UNKNOWN } } }
I20260927 05:17:34.788411 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.783217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.97.1:40487)
0927 05:17:34.783450 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.783454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.783474 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:17:34.783639 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.783644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.783740 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.783866 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:34.784218 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.784920 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.786280 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.786701 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.786805 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.787324 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:34.787801 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:34.787933 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.788048 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.788115 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:17:34.796172 19036 raft_consensus.cc:1261] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 1 FOLLOWER]: Refusing update from remote peer 790b37f30c034dbdb6490e3dccc043c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:34.796403 18903 raft_consensus.cc:1261] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 1 FOLLOWER]: Refusing update from remote peer 790b37f30c034dbdb6490e3dccc043c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:34.797147 19110 consensus_queue.cc:1035] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eff2436753f43e8b5cca0a99ecb5a10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:34.797494 19107 consensus_queue.cc:1035] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "991966b2845f4247a02deaf595ad2706" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:34.808630 19113 mvcc.cc:205] Tried to move back new op lower bound from 7333831699635834880 to 7333831699241033728. Current Snapshot: MvccSnapshot[applied={T|T < 7333831699635834880}]
I20260927 05:17:34.823607 18625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:34.823890 18625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:34.827379 18625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4660a5244ddd4e55a0ca2ffc8e65d7be: Sending DeleteTablet for 3 replicas of tablet cf52afaab1b749ea95c3d90cf3812c64
I20260927 05:17:34.828431 18883 tablet_service.cc:1514] Processing DeleteTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 05:17:34.828806 18749 tablet_service.cc:1514] Processing DeleteTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49266
I20260927 05:17:34.829248 19016 tablet_service.cc:1514] Processing DeleteTablet for tablet cf52afaab1b749ea95c3d90cf3812c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46486
I20260927 05:17:34.832494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18686
I20260927 05:17:34.867668 19124 tablet_replica.cc:330] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: stopping tablet replica
I20260927 05:17:34.868290 19124 raft_consensus.cc:2229] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:34.869091 19124 raft_consensus.cc:2258] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:34.874850 19124 ts_tablet_manager.cc:1905] T cf52afaab1b749ea95c3d90cf3812c64 P 790b37f30c034dbdb6490e3dccc043c3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:17:34 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:17:34.879675 19123 tablet_replica.cc:330] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: stopping tablet replica
I20260927 05:17:34.880761 19125 tablet_replica.cc:330] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: stopping tablet replica
I20260927 05:17:34.881114 19125 raft_consensus.cc:2229] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:34.881518 19125 raft_consensus.cc:2258] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:34.882081 19123 raft_consensus.cc:2229] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:34.882503 19123 raft_consensus.cc:2258] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:34.884135 19125 ts_tablet_manager.cc:1905] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:34.885842 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:34.837283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.0.97.1:40487)
0927 05:17:34.837497 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:34.837506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:34.858590 (+ 21084us) server_negotiation.cc:407] Connection header received
0927 05:17:34.859184 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:34.859193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:34.859299 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:34.859396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:34.861653 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.863293 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.871080 (+ 7787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:34.871791 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:34.871920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:34.878048 (+ 6128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:34.878067 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:34.878086 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:34.878132 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:34.882588 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.883471 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.883479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.883482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.883670 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:34.884107 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:34.884112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:34.884116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:34.884549 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:34.884839 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:34.885205 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 05:17:34.885433 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:17:34.886549 18809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:34.887208 19123 ts_tablet_manager.cc:1905] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:34.893783 19125 ts_tablet_manager.cc:1918] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:34.893970 19125 log.cc:1196] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/cf52afaab1b749ea95c3d90cf3812c64
I20260927 05:17:34.894425 19125 ts_tablet_manager.cc:1939] T cf52afaab1b749ea95c3d90cf3812c64 P 8eff2436753f43e8b5cca0a99ecb5a10: Deleting consensus metadata
I20260927 05:17:34.895285 19123 ts_tablet_manager.cc:1918] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:34.895449 19123 log.cc:1196] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/cf52afaab1b749ea95c3d90cf3812c64
I20260927 05:17:34.895944 19123 ts_tablet_manager.cc:1939] T cf52afaab1b749ea95c3d90cf3812c64 P 991966b2845f4247a02deaf595ad2706: Deleting consensus metadata
I20260927 05:17:34.896154 18610 catalog_manager.cc:4841] TS 8eff2436753f43e8b5cca0a99ecb5a10 (127.0.97.3:34891): tablet cf52afaab1b749ea95c3d90cf3812c64 (table test-table [id=9ec3d9858dcb40a4b05190a702964894]) successfully deleted
I20260927 05:17:34.897569 18612 catalog_manager.cc:4841] TS 991966b2845f4247a02deaf595ad2706 (127.0.97.2:42891): tablet cf52afaab1b749ea95c3d90cf3812c64 (table test-table [id=9ec3d9858dcb40a4b05190a702964894]) successfully deleted
I20260927 05:17:34.988495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18686
W20260927 05:17:35.007915 18611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40487 (error 108)
I20260927 05:17:35.008088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18820
I20260927 05:17:35.009976 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.009166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49296)
0927 05:17:35.009600 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.009735 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":190,"threads_started":1}
W20260927 05:17:35.010206 18611 catalog_manager.cc:4584] TS 790b37f30c034dbdb6490e3dccc043c3 (127.0.97.1:40487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf52afaab1b749ea95c3d90cf3812c64: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Sep 27 05:17:35 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:17:35.022658 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.008859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.97.2:42891)
0927 05:17:35.009071 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:35.009078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:35.009098 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:17:35.009260 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:35.009268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:35.009373 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:35.009494 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:35.010989 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.011758 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.012961 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.013386 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.013476 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:35.017661 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:35.017675 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:35.017690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:35.017721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:35.020349 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:35.020966 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:35.020972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:35.020975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:35.021092 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:35.021385 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:35.021388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:35.021390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:35.021723 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:35.021949 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:35.022145 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 05:17:35.022297 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:17:35.023427 18943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:35.063961 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.063425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49298)
0927 05:17:35.063652 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.063788 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:17:35.065034 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.064617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49306)
0927 05:17:35.064789 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.064870 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:17:35.112437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18820
I20260927 05:17:35.125423 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.124987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49318)
0927 05:17:35.125179 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.125284 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 05:17:35.126504 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.126090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49334)
0927 05:17:35.126233 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.126348 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 05:17:35.130034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18953
Sep 27 05:17:35 dist-test-slave-1t6p krb5kdc[18580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:17:35.144025 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.130688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.97.3:34891)
0927 05:17:35.130846 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:35.130851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:35.130870 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:17:35.130984 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:35.130988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:35.131105 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:35.131184 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:35.132407 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.133138 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.134383 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.134743 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.134827 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:35.138903 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:35.138916 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:35.138930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:35.138964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:35.141764 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:35.142360 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:35.142367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:35.142369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:35.142493 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:35.142811 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:35.142814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:35.142816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:35.143198 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:35.143418 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:35.143649 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 05:17:35.143798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:17:35.144663 19076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:35.207075 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.206498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49340)
0927 05:17:35.206797 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.206879 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 05:17:35.208108 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.207723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40487 (local address 127.0.0.1:49344)
0927 05:17:35.207912 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:35.207985 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 05:17:35.231184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18953
I20260927 05:17:35.247563 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 18593
I20260927 05:17:35.248592 18655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:35.355665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 18593
2026-09-27T05:17:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4447 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T05:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:35Z Disabled control of system clock
I20260927 05:17:35.427985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42325
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:35.746156 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:35.746412 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:35.746488 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:35.753361 19136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:35.753448 19136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:35.753484 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:35.753533 19136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:35.753574 19136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:35.762279 19136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:35.763471 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:35.765102 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:35.776767 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:35.776821 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:35.777688 19144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:35.778170 19136 server_base.cc:1034] running on GCE node
I20260927 05:17:35.778878 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:35.780171 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:35.781359 19136 hybrid_clock.cc:648] HybridClock initialized: now 1790486255781302 us; error 47 us; skew 500 ppm
I20260927 05:17:35.781683 19136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:35.783824 19136 webserver.cc:460] Webserver started at http://127.0.97.62:36813/ using document root <none> and password file <none>
I20260927 05:17:35.784461 19136 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:35.784543 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:35.784884 19136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:35.787657 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "4b5787895aad49fa8479d57474da9cce"
format_stamp: "Formatted at 2026-09-27 05:17:35 on dist-test-slave-1t6p"
server_key: "00aaa3cfd1a3436a66dd042f13d4a88d"
server_key_iv: "10e8eaca99adc4c9c5210acb55ce095e"
server_key_version: "encryptionkey@0"
I20260927 05:17:35.788426 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b5787895aad49fa8479d57474da9cce"
format_stamp: "Formatted at 2026-09-27 05:17:35 on dist-test-slave-1t6p"
server_key: "00aaa3cfd1a3436a66dd042f13d4a88d"
server_key_iv: "10e8eaca99adc4c9c5210acb55ce095e"
server_key_version: "encryptionkey@0"
I20260927 05:17:35.793069 19136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:35.796286 19150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:35.797483 19136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:35.797650 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "4b5787895aad49fa8479d57474da9cce"
format_stamp: "Formatted at 2026-09-27 05:17:35 on dist-test-slave-1t6p"
server_key: "00aaa3cfd1a3436a66dd042f13d4a88d"
server_key_iv: "10e8eaca99adc4c9c5210acb55ce095e"
server_key_version: "encryptionkey@0"
I20260927 05:17:35.797849 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:35.810388 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:35.811153 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:35.811424 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:35.830019 19136 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42325
I20260927 05:17:35.830052 19201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42325 every 8 connection(s)
I20260927 05:17:35.831349 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:35.834791 19202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:35.836644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19136
I20260927 05:17:35.836818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:35.837409 388 external_mini_cluster.cc:1428] Setting key 2a8089e5fb8969404cf72e0539fe82a7
I20260927 05:17:35.845986 19202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: Bootstrap starting.
I20260927 05:17:35.849020 19202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:35.850036 19202 log.cc:824] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:35.850261 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:35.839377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.0.97.62:42325)
0927 05:17:35.840386 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:35.840396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:35.840432 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:17:35.842014 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:35.842034 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:35.842435 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:35.842790 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:35.843362 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.844614 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.845960 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:35.846676 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:35.846891 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:35.848098 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:35.848106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:35.848118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:35.848600 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:35.848682 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:35.848701 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:35.848822 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:35.849064 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 05:17:35.849376 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
I20260927 05:17:35.852833 19202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: No bootstrap required, opened a new log
I20260927 05:17:35.859790 19202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } }
I20260927 05:17:35.860121 19202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:35.860235 19202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5787895aad49fa8479d57474da9cce, State: Initialized, Role: FOLLOWER
I20260927 05:17:35.860782 19202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } }
I20260927 05:17:35.861002 19202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:35.861168 19202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:35.861325 19202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:35.863572 19202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } }
I20260927 05:17:35.864008 19202 leader_election.cc:302] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b5787895aad49fa8479d57474da9cce; no voters:
I20260927 05:17:35.864529 19202 leader_election.cc:288] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:35.864686 19207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:35.865458 19207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [term 1 LEADER]: Becoming Leader. State: Replica: 4b5787895aad49fa8479d57474da9cce, State: Running, Role: LEADER
I20260927 05:17:35.865960 19207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } }
I20260927 05:17:35.866626 19202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:35.868837 19208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b5787895aad49fa8479d57474da9cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } } }
I20260927 05:17:35.869252 19208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:35.869869 19209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5787895aad49fa8479d57474da9cce. Latest consensus state: current_term: 1 leader_uuid: "4b5787895aad49fa8479d57474da9cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5787895aad49fa8479d57474da9cce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42325 } } }
I20260927 05:17:35.870169 19209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:35.870223 19216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:35.875454 19216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:35.883945 19216 catalog_manager.cc:1347] Generated new cluster ID: ca2566963c2d435f89b2c8bc4a37569e
I20260927 05:17:35.884063 19216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:35.894308 19216 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:35.896127 19216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:35.906083 19216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: Generated new TSK 0
I20260927 05:17:35.907051 19216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:35.917253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--builtin_ntp_servers=127.0.97.20:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:36.237452 19226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:36.237744 19226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:36.237906 19226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:36.244699 19226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:36.244817 19226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:36.244997 19226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:36.253468 19226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:36.254595 19226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:36.256313 19226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:36.269729 19231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:36.269754 19234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:36.270313 19226 server_base.cc:1034] running on GCE node
W20260927 05:17:36.270398 19232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:36.271131 19226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:36.272037 19226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:36.273263 19226 hybrid_clock.cc:648] HybridClock initialized: now 1790486256273229 us; error 53 us; skew 500 ppm
I20260927 05:17:36.273567 19226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:36.275651 19226 webserver.cc:460] Webserver started at http://127.0.97.1:34193/ using document root <none> and password file <none>
I20260927 05:17:36.276292 19226 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:36.276418 19226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:36.276722 19226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:36.279476 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "7201c059a1bd42da82634d694ce0540a"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "3bb4504b3b6b5fd1155cdb4530490ed9"
server_key_iv: "b75c5e64fa3709abb2c3dc938e5721ab"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.280220 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7201c059a1bd42da82634d694ce0540a"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "3bb4504b3b6b5fd1155cdb4530490ed9"
server_key_iv: "b75c5e64fa3709abb2c3dc938e5721ab"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.284900 19226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:36.288214 19240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.289414 19226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:36.289572 19226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "7201c059a1bd42da82634d694ce0540a"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "3bb4504b3b6b5fd1155cdb4530490ed9"
server_key_iv: "b75c5e64fa3709abb2c3dc938e5721ab"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.289780 19226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:36.304174 19226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:36.305243 19226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:36.305531 19226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:36.306404 19226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:36.307817 19226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:36.307946 19226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.308063 19226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:36.308169 19226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.338440 19226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45859
I20260927 05:17:36.338482 19352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45859 every 8 connection(s)
I20260927 05:17:36.339781 19226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:36.341948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19226
I20260927 05:17:36.342157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:36.342680 388 external_mini_cluster.cc:1428] Setting key 119e7a61114175fb3f76f16f1a6324f3
I20260927 05:17:36.345520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--builtin_ntp_servers=127.0.97.20:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:36.353737 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:36.342348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48645 (local address 127.0.97.62:42325)
0927 05:17:36.342559 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:36.342566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:36.344195 (+ 1629us) server_negotiation.cc:407] Connection header received
0927 05:17:36.346165 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:36.346174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:36.346264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:36.346377 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:36.347406 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.348510 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.350384 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.351170 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.351309 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:36.352779 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:36.352788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:36.352796 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:36.352820 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:36.352864 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:36.352881 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:36.352977 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:36.353236 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 05:17:36.353358 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:17:36.355691 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:36.342957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42325 (local address 127.0.97.1:48645)
0927 05:17:36.344013 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:36.344066 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:36.345894 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:36.346557 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:36.346564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:36.346644 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:36.347239 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:36.347257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.348653 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.348656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:36.350212 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:36.350228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.351379 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.351387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:36.351671 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:36.352438 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:36.352443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:36.352490 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:36.352577 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:36.352982 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:36.352993 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:36.353003 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:17:36.353995 (+ 992us) client_negotiation.cc:240] Negotiation successful
0927 05:17:36.354385 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":300,"threads_started":1}
I20260927 05:17:36.357259 19353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42325
I20260927 05:17:36.357563 19353 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:36.358390 19353 heartbeater.cc:511] Master 127.0.97.62:42325 requested a full tablet report, sending...
I20260927 05:17:36.360471 19167 ts_manager.cc:194] Registered new tserver with Master: 7201c059a1bd42da82634d694ce0540a (127.0.97.1:45859)
I20260927 05:17:36.361904 19167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48645
W20260927 05:17:36.675043 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:36.675324 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:36.675482 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:36.682435 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:36.682552 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:36.682750 19356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:36.691282 19356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:36.692448 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:36.694240 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:36.706941 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:36.706985 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:36.707072 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:36.707576 19356 server_base.cc:1034] running on GCE node
I20260927 05:17:36.708139 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:36.708981 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:36.710201 19356 hybrid_clock.cc:648] HybridClock initialized: now 1790486256710148 us; error 63 us; skew 500 ppm
I20260927 05:17:36.710500 19356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:36.712667 19356 webserver.cc:460] Webserver started at http://127.0.97.2:32839/ using document root <none> and password file <none>
I20260927 05:17:36.713295 19356 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:36.713418 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:36.713744 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:36.716389 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "87b56ea8d7157cd59e95bfbf1931b778"
server_key_iv: "7caa4108d48274a26625dbe87a2505ea"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.717123 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "87b56ea8d7157cd59e95bfbf1931b778"
server_key_iv: "7caa4108d48274a26625dbe87a2505ea"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.721886 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:17:36.724943 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.726063 19356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:17:36.726223 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1"
format_stamp: "Formatted at 2026-09-27 05:17:36 on dist-test-slave-1t6p"
server_key: "87b56ea8d7157cd59e95bfbf1931b778"
server_key_iv: "7caa4108d48274a26625dbe87a2505ea"
server_key_version: "encryptionkey@0"
I20260927 05:17:36.726429 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:36.737151 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:36.737861 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:36.738126 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:36.739068 19356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:36.740504 19356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:36.740619 19356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.740751 19356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:36.740839 19356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:36.773136 19356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34331
I20260927 05:17:36.773191 19482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34331 every 8 connection(s)
I20260927 05:17:36.774397 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:36.780555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19356
I20260927 05:17:36.780737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:36.781200 388 external_mini_cluster.cc:1428] Setting key ad9f4482fd3f56ffb4bf9595331b9d52
I20260927 05:17:36.783749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--builtin_ntp_servers=127.0.97.20:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:36.787376 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:36.776824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49589 (local address 127.0.97.62:42325)
0927 05:17:36.777003 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:36.777008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:36.778348 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 05:17:36.780227 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:36.780233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:36.780329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:36.780440 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:36.781584 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.782434 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.784077 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.784828 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.784975 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:36.786446 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:36.786456 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:36.786464 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:36.786487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:36.786529 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:36.786547 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:36.786644 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:36.786834 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 05:17:36.786947 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:17:36.788225 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:36.777338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42325 (local address 127.0.97.2:49589)
0927 05:17:36.778190 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:36.778233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:36.779949 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:36.780664 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:36.780693 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:36.780824 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:36.781429 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:36.781447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.782580 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.782585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:36.783886 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:36.783903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:36.785083 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:36.785089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:36.785282 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:36.786152 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:36.786155 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:36.786192 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:36.786259 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:36.786646 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:36.786654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:36.786659 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:17:36.786799 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 05:17:36.787129 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260927 05:17:36.789583 19483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42325
I20260927 05:17:36.789871 19483 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:36.790514 19483 heartbeater.cc:511] Master 127.0.97.62:42325 requested a full tablet report, sending...
I20260927 05:17:36.791906 19167 ts_manager.cc:194] Registered new tserver with Master: 1dd50ef63c4e4930bc3d8cf307cdccd1 (127.0.97.2:34331)
I20260927 05:17:36.792717 19167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49589
W20260927 05:17:37.102859 19486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:37.103156 19486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:37.103320 19486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:37.110169 19486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:37.110251 19486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:37.110414 19486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:37.119318 19486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:37.120456 19486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:37.122113 19486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:37.134768 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:37.134660 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:37.134732 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:37.135890 19486 server_base.cc:1034] running on GCE node
I20260927 05:17:37.136449 19486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:37.137406 19486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:37.138641 19486 hybrid_clock.cc:648] HybridClock initialized: now 1790486257138605 us; error 41 us; skew 500 ppm
I20260927 05:17:37.138952 19486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:37.141120 19486 webserver.cc:460] Webserver started at http://127.0.97.3:33095/ using document root <none> and password file <none>
I20260927 05:17:37.141779 19486 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:37.141911 19486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:37.142232 19486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:37.144994 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "93721010ec48425290f49afeebc29e98"
format_stamp: "Formatted at 2026-09-27 05:17:37 on dist-test-slave-1t6p"
server_key: "706ae891cba2a489497676971d6ea359"
server_key_iv: "4dbc89cac96ec2b6edbeeecfae38e001"
server_key_version: "encryptionkey@0"
I20260927 05:17:37.145787 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "93721010ec48425290f49afeebc29e98"
format_stamp: "Formatted at 2026-09-27 05:17:37 on dist-test-slave-1t6p"
server_key: "706ae891cba2a489497676971d6ea359"
server_key_iv: "4dbc89cac96ec2b6edbeeecfae38e001"
server_key_version: "encryptionkey@0"
I20260927 05:17:37.150494 19486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 05:17:37.153781 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:37.154987 19486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:17:37.155229 19486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "93721010ec48425290f49afeebc29e98"
format_stamp: "Formatted at 2026-09-27 05:17:37 on dist-test-slave-1t6p"
server_key: "706ae891cba2a489497676971d6ea359"
server_key_iv: "4dbc89cac96ec2b6edbeeecfae38e001"
server_key_version: "encryptionkey@0"
I20260927 05:17:37.155437 19486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:37.173966 19486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:37.174703 19486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:37.174968 19486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:37.175895 19486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:37.177265 19486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:37.177386 19486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:37.177523 19486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:37.177611 19486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:37.210141 19486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42081
I20260927 05:17:37.210199 19612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42081 every 8 connection(s)
I20260927 05:17:37.211462 19486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:37.218292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19486
I20260927 05:17:37.218467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:37.219022 388 external_mini_cluster.cc:1428] Setting key 5a40c2bbe1888ea3635c5cbd37448973
I20260927 05:17:37.224010 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.213879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56731 (local address 127.0.97.62:42325)
0927 05:17:37.214083 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.214090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.215604 (+ 1514us) server_negotiation.cc:407] Connection header received
0927 05:17:37.217306 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.217311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.217375 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.217458 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:37.218457 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.219553 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.221409 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.221897 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.222050 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.223385 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:37.223389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:37.223392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:37.223408 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:37.223438 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:37.223449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:37.223517 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.223701 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.223785 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:17:37.224923 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.214425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42325 (local address 127.0.97.3:56731)
0927 05:17:37.215468 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.215510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.217069 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.217645 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.217653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.217732 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:37.218276 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.218291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.219770 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.219777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.221284 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.221295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.222144 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.222149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.222342 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.223104 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:37.223107 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:37.223144 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:37.223223 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:37.223533 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:37.223540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:37.223546 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.223678 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.223959 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
I20260927 05:17:37.226112 19613 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42325
I20260927 05:17:37.226402 19613 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:37.227156 19613 heartbeater.cc:511] Master 127.0.97.62:42325 requested a full tablet report, sending...
I20260927 05:17:37.228392 19167 ts_manager.cc:194] Registered new tserver with Master: 93721010ec48425290f49afeebc29e98 (127.0.97.3:42081)
I20260927 05:17:37.229175 19167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56731
I20260927 05:17:37.233966 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:37.243779 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.237956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.0.97.62:42325)
0927 05:17:37.238121 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.238128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.238500 (+ 372us) server_negotiation.cc:407] Connection header received
0927 05:17:37.238668 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.238673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.238775 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.238877 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:37.239355 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.240050 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.241360 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.241742 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.241877 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.243059 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:37.243064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:37.243068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:37.243084 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:37.243111 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:37.243123 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:37.243275 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.243467 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.243515 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:17:37.247098 19167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:37.253571 19167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:37.277117 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.264652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.0.1:36560)
0927 05:17:37.265696 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.265734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.265892 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.266960 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.266971 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.267038 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:37.267588 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.267604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.270334 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.270342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.273390 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.273410 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.275932 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.275938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.276216 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.276291 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.276413 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.276661 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":508,"threads_started":1}
I20260927 05:17:37.280076 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.264927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.0.97.2:34331)
0927 05:17:37.266273 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.266282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.266315 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:17:37.266420 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.266428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.266676 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.267842 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:37.268150 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.270148 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.273684 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.275637 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.277724 (+ 2087us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.278048 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.279493 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.279707 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":737,"threads_started":1}
I20260927 05:17:37.284011 19418 tablet_service.cc:1467] Processing CreateTablet for tablet e79a007914c4428cab632470beff4721 (DEFAULT_TABLE table=test-table [id=f1b589be41b54ef7a76f83d22c2d9388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:37.285537 19418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a007914c4428cab632470beff4721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:37.287468 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.266518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42081 (local address 127.0.0.1:40026)
0927 05:17:37.277444 (+ 10926us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.277460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.277544 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.280334 (+ 2790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.280342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.280377 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:37.280636 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.280649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.283793 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.283798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.285382 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.285407 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.286657 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.286662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.286833 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.286865 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.286945 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.287098 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10840,"mutex_wait_us":558,"thread_start_us":16455,"threads_started":1}
I20260927 05:17:37.287901 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.266971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.0.97.3:42081)
0927 05:17:37.279403 (+ 12432us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.279415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.279459 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:17:37.279576 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.279586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.279863 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.280843 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:37.281159 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.283218 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.285550 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.286502 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.286647 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.286910 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.286989 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.287549 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12205,"thread_start_us":13043,"threads_started":1}
I20260927 05:17:37.288138 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.265987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40920)
0927 05:17:37.266663 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.266680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.266804 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.280789 (+ 13985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.280793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.280818 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:37.281259 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.281273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.283288 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.283293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.285509 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.285526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.287364 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.287374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.287593 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.287623 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.287688 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.287800 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":4,"thread_start_us":615,"threads_started":1}
I20260927 05:17:37.289276 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.266451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.97.1:45859)
0927 05:17:37.279889 (+ 13438us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.279901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.279931 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:17:37.280029 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.280038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.280584 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.281479 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:37.281679 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.283131 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.285749 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.287135 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.288514 (+ 1379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.288723 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.288783 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.288936 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13267,"thread_start_us":13800,"threads_started":1}
I20260927 05:17:37.292022 19548 tablet_service.cc:1467] Processing CreateTablet for tablet e79a007914c4428cab632470beff4721 (DEFAULT_TABLE table=test-table [id=f1b589be41b54ef7a76f83d22c2d9388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:37.292528 19288 tablet_service.cc:1467] Processing CreateTablet for tablet e79a007914c4428cab632470beff4721 (DEFAULT_TABLE table=test-table [id=f1b589be41b54ef7a76f83d22c2d9388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:37.293519 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a007914c4428cab632470beff4721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:37.293915 19288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a007914c4428cab632470beff4721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:37.301131 19631 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Bootstrap starting.
I20260927 05:17:37.304481 19631 tablet_bootstrap.cc:654] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:37.305433 19631 log.cc:824] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:37.310653 19633 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Bootstrap starting.
I20260927 05:17:37.316258 19634 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Bootstrap starting.
I20260927 05:17:37.316586 19633 tablet_bootstrap.cc:654] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:37.317459 19631 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: No bootstrap required, opened a new log
I20260927 05:17:37.317719 19631 ts_tablet_manager.cc:1397] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260927 05:17:37.317816 19633 log.cc:824] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:37.319595 19634 tablet_bootstrap.cc:654] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:37.320652 19633 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: No bootstrap required, opened a new log
I20260927 05:17:37.320652 19634 log.cc:824] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:37.320912 19633 ts_tablet_manager.cc:1397] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 05:17:37.322983 19634 tablet_bootstrap.cc:492] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: No bootstrap required, opened a new log
I20260927 05:17:37.323278 19634 ts_tablet_manager.cc:1397] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 05:17:37.327893 19631 raft_consensus.cc:348] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.328315 19631 raft_consensus.cc:374] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:37.328449 19631 raft_consensus.cc:729] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd50ef63c4e4930bc3d8cf307cdccd1, State: Initialized, Role: FOLLOWER
I20260927 05:17:37.329034 19631 consensus_queue.cc:260] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.331501 19483 heartbeater.cc:503] Master 127.0.97.62:42325 was elected leader, sending a full tablet report...
I20260927 05:17:37.331310 19634 raft_consensus.cc:348] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.331321 19633 raft_consensus.cc:348] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.331709 19634 raft_consensus.cc:374] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:37.331715 19633 raft_consensus.cc:374] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:37.331799 19634 raft_consensus.cc:729] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93721010ec48425290f49afeebc29e98, State: Initialized, Role: FOLLOWER
I20260927 05:17:37.331825 19633 raft_consensus.cc:729] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201c059a1bd42da82634d694ce0540a, State: Initialized, Role: FOLLOWER
I20260927 05:17:37.332123 19631 ts_tablet_manager.cc:1428] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 05:17:37.332371 19633 consensus_queue.cc:260] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.332440 19634 consensus_queue.cc:260] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.335968 19353 heartbeater.cc:503] Master 127.0.97.62:42325 was elected leader, sending a full tablet report...
I20260927 05:17:37.336158 19634 ts_tablet_manager.cc:1428] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260927 05:17:37.336624 19613 heartbeater.cc:503] Master 127.0.97.62:42325 was elected leader, sending a full tablet report...
I20260927 05:17:37.336650 19633 ts_tablet_manager.cc:1428] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
W20260927 05:17:37.341925 19354 tablet.cc:2369] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:37.463225 19614 tablet.cc:2369] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:37.526352 19484 tablet.cc:2369] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:37.568195 19639 raft_consensus.cc:482] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:37.568472 19639 raft_consensus.cc:504] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.570303 19639 leader_election.cc:288] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd50ef63c4e4930bc3d8cf307cdccd1 (127.0.97.2:34331), 7201c059a1bd42da82634d694ce0540a (127.0.97.1:45859)
I20260927 05:17:37.576705 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.570642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33421 (local address 127.0.97.2:34331)
0927 05:17:37.570824 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.570832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.570853 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:17:37.571586 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.571592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.571671 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.571763 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:37.572276 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.573208 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.575074 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.575810 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.575952 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.576049 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.576332 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.576458 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 05:17:37.576764 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.570523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.97.3:33421)
0927 05:17:37.570760 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.570781 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.570919 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.571852 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.571857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.571889 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:37.572111 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.572121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.573339 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.573344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.574891 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.574902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.575957 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.575963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.576155 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.576178 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.576302 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.576433 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":42}
I20260927 05:17:37.577155 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.571290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36985 (local address 127.0.97.1:45859)
0927 05:17:37.571501 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.571506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.571758 (+ 252us) server_negotiation.cc:407] Connection header received
0927 05:17:37.572032 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.572038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.572137 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.572234 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:37.572859 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.573748 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.575562 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.576309 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.576448 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.576535 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.576795 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.576915 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:17:37.577230 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.571002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.97.3:36985)
0927 05:17:37.571661 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.571673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:37.571821 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:37.572292 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:37.572296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:37.572319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:37.572689 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.572700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.573902 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.573911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.575389 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:37.575399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.576475 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.576481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:37.576644 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.576665 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 05:17:37.576763 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:17:37.576887 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":3,"thread_start_us":324,"threads_started":1}
I20260927 05:17:37.577687 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a007914c4428cab632470beff4721" candidate_uuid: "93721010ec48425290f49afeebc29e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" is_pre_election: true
I20260927 05:17:37.578171 19438 raft_consensus.cc:2454] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93721010ec48425290f49afeebc29e98 in term 0.
I20260927 05:17:37.578826 19502 leader_election.cc:302] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd50ef63c4e4930bc3d8cf307cdccd1, 93721010ec48425290f49afeebc29e98; no voters:
I20260927 05:17:37.578779 19308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a007914c4428cab632470beff4721" candidate_uuid: "93721010ec48425290f49afeebc29e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201c059a1bd42da82634d694ce0540a" is_pre_election: true
I20260927 05:17:37.579195 19639 raft_consensus.cc:2781] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:37.579342 19639 raft_consensus.cc:482] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:37.579322 19308 raft_consensus.cc:2454] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93721010ec48425290f49afeebc29e98 in term 0.
I20260927 05:17:37.579448 19639 raft_consensus.cc:3037] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:37.581653 19639 raft_consensus.cc:504] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.582116 19639 leader_election.cc:288] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [CANDIDATE]: Term 1 election: Requested vote from peers 1dd50ef63c4e4930bc3d8cf307cdccd1 (127.0.97.2:34331), 7201c059a1bd42da82634d694ce0540a (127.0.97.1:45859)
I20260927 05:17:37.582536 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a007914c4428cab632470beff4721" candidate_uuid: "93721010ec48425290f49afeebc29e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1"
I20260927 05:17:37.582614 19308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a007914c4428cab632470beff4721" candidate_uuid: "93721010ec48425290f49afeebc29e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201c059a1bd42da82634d694ce0540a"
I20260927 05:17:37.582731 19438 raft_consensus.cc:3037] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:37.582830 19308 raft_consensus.cc:3037] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:37.584875 19438 raft_consensus.cc:2454] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93721010ec48425290f49afeebc29e98 in term 1.
I20260927 05:17:37.584944 19308 raft_consensus.cc:2454] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93721010ec48425290f49afeebc29e98 in term 1.
I20260927 05:17:37.585265 19502 leader_election.cc:302] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd50ef63c4e4930bc3d8cf307cdccd1, 93721010ec48425290f49afeebc29e98; no voters:
I20260927 05:17:37.585544 19639 raft_consensus.cc:2781] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:37.586002 19639 raft_consensus.cc:686] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 1 LEADER]: Becoming Leader. State: Replica: 93721010ec48425290f49afeebc29e98, State: Running, Role: LEADER
I20260927 05:17:37.586457 19639 consensus_queue.cc:237] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } }
I20260927 05:17:37.590869 19166 catalog_manager.cc:5495] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93721010ec48425290f49afeebc29e98 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "93721010ec48425290f49afeebc29e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93721010ec48425290f49afeebc29e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42081 } health_report { overall_health: HEALTHY } } }
I20260927 05:17:37.655205 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.649678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.97.3:42081)
0927 05:17:37.649905 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.649912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.649938 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:17:37.650039 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.650044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.650142 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.650297 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:37.650728 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.651499 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.652891 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.653343 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.653481 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.654061 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:37.654535 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:37.654721 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.654889 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.654933 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:17:37.663305 19438 raft_consensus.cc:1261] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 1 FOLLOWER]: Refusing update from remote peer 93721010ec48425290f49afeebc29e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:37.663599 19308 raft_consensus.cc:1261] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 1 FOLLOWER]: Refusing update from remote peer 93721010ec48425290f49afeebc29e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:37.664647 19641 consensus_queue.cc:1035] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201c059a1bd42da82634d694ce0540a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:37.665022 19639 consensus_queue.cc:1035] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd50ef63c4e4930bc3d8cf307cdccd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:37.686697 19166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:37.687042 19166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:37.691541 19166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b5787895aad49fa8479d57474da9cce: Sending DeleteTablet for 3 replicas of tablet e79a007914c4428cab632470beff4721
I20260927 05:17:37.692858 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a007914c4428cab632470beff4721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:37 UTC) from {username='slave'} at 127.0.0.1:40026
I20260927 05:17:37.693902 19654 tablet_replica.cc:330] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: stopping tablet replica
I20260927 05:17:37.694067 19418 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a007914c4428cab632470beff4721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:37 UTC) from {username='slave'} at 127.0.0.1:36560
I20260927 05:17:37.694371 19654 raft_consensus.cc:2229] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:37.694969 19288 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a007914c4428cab632470beff4721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:37 UTC) from {username='slave'} at 127.0.0.1:40920
I20260927 05:17:37.695055 19654 raft_consensus.cc:2258] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:37.697234 19654 ts_tablet_manager.cc:1905] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:37.698902 19656 tablet_replica.cc:330] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: stopping tablet replica
I20260927 05:17:37.700073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19226
I20260927 05:17:37.708804 19654 ts_tablet_manager.cc:1918] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:37.709003 19654 log.cc:1196] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/e79a007914c4428cab632470beff4721
I20260927 05:17:37.709664 19654 ts_tablet_manager.cc:1939] T e79a007914c4428cab632470beff4721 P 93721010ec48425290f49afeebc29e98: Deleting consensus metadata
I20260927 05:17:37.711688 19151 catalog_manager.cc:4841] TS 93721010ec48425290f49afeebc29e98 (127.0.97.3:42081): tablet e79a007914c4428cab632470beff4721 (table test-table [id=f1b589be41b54ef7a76f83d22c2d9388]) successfully deleted
I20260927 05:17:37.712483 19655 tablet_replica.cc:330] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: stopping tablet replica
I20260927 05:17:37.712960 19655 raft_consensus.cc:2229] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:37.713524 19655 raft_consensus.cc:2258] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:37.715237 19656 raft_consensus.cc:2229] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:37.716125 19656 raft_consensus.cc:2258] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:37.716848 19655 ts_tablet_manager.cc:1905] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:37.719767 19656 ts_tablet_manager.cc:1905] T e79a007914c4428cab632470beff4721 P 7201c059a1bd42da82634d694ce0540a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:37.723469 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.701523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.97.1:45859)
0927 05:17:37.701685 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.701691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.714414 (+ 12723us) server_negotiation.cc:407] Connection header received
0927 05:17:37.714620 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.714626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.714737 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.714850 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:37.715563 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.716820 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.719611 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.720278 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.720377 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.721818 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:37.721828 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:37.721844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:37.722545 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:37.722600 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:37.722623 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:37.722717 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.722940 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.723090 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:17:37.723489 19655 ts_tablet_manager.cc:1918] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:37.723634 19655 log.cc:1196] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/e79a007914c4428cab632470beff4721
I20260927 05:17:37.724146 19655 ts_tablet_manager.cc:1939] T e79a007914c4428cab632470beff4721 P 1dd50ef63c4e4930bc3d8cf307cdccd1: Deleting consensus metadata
I20260927 05:17:37.724534 19348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:37.725914 19152 catalog_manager.cc:4841] TS 1dd50ef63c4e4930bc3d8cf307cdccd1 (127.0.97.2:34331): tablet e79a007914c4428cab632470beff4721 (table test-table [id=f1b589be41b54ef7a76f83d22c2d9388]) successfully deleted
I20260927 05:17:37.828471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19226
W20260927 05:17:37.852454 19152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45859 (error 108)
I20260927 05:17:37.852622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19356
I20260927 05:17:37.854566 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.853738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40928)
0927 05:17:37.854211 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.854377 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
W20260927 05:17:37.854846 19152 catalog_manager.cc:4584] TS 7201c059a1bd42da82634d694ce0540a (127.0.97.1:45859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e79a007914c4428cab632470beff4721: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
I20260927 05:17:37.860836 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.853794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.0.97.2:34331)
0927 05:17:37.853992 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:37.854000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:37.854024 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:17:37.854090 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:37.854095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:37.854198 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:37.854311 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:37.854844 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.855711 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.857585 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:37.857974 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:37.858057 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:37.859438 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:37.859446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:37.859459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:37.860123 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:37.860175 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:37.860194 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:37.860314 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:37.860428 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 05:17:37.860508 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:17:37.861433 19478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:37.908479 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.908032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40936)
0927 05:17:37.908242 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.908337 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:17:37.909632 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.909254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40948)
0927 05:17:37.909436 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.909499 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:17:37.970170 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.969665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40960)
0927 05:17:37.969883 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.969985 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:17:37.971334 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:37.970901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40966)
0927 05:17:37.971080 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:37.971168 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:17:37.988111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19356
I20260927 05:17:38.005453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19486
I20260927 05:17:38.012735 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:38.006277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.0.97.3:42081)
0927 05:17:38.006451 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:38.006459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:38.006483 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:17:38.006712 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:38.006718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:38.006835 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:38.006958 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:38.007379 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:38.008200 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:38.009543 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:38.010011 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:38.010157 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:38.011424 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:38.011433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:38.011445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:38.011995 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:38.012038 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:38.012057 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:38.012176 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:38.012311 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 05:17:38.012412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:17:38.013401 19608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:38.052737 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:38.052159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40974)
0927 05:17:38.052443 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:38.052552 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 05:17:38.053889 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:38.053436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40988)
0927 05:17:38.053650 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:38.053742 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 05:17:38.130141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19486
I20260927 05:17:38.149287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19136
I20260927 05:17:38.150367 19197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:38.256837 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:38.256063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45859 (local address 127.0.0.1:40996)
0927 05:17:38.256497 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:38.256629 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260927 05:17:38.256826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19136
2026-09-27T05:17:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2899 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:38 dist-test-slave-1t6p krb5kdc[19666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:38 dist-test-slave-1t6p krb5kdc[19666](info): set up 2 sockets
Sep 27 05:17:38 dist-test-slave-1t6p krb5kdc[19666](info): commencing operation
krb5kdc: starting...
W20260927 05:17:39.367246 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:39 dist-test-slave-1t6p krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:17:39.588302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34095
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:39.905360 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:39.905625 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:39.905846 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:39.912819 19679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:39.912912 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:39.912999 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:39.913087 19679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:39.913169 19679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:39.922416 19679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:39.923642 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:39.924928 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:39.936594 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:39.936630 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:39.936734 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:39.937517 19679 server_base.cc:1034] running on GCE node
I20260927 05:17:39.938269 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:39.939608 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:39.940840 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790486259940798 us; error 54 us; skew 500 ppm
I20260927 05:17:39.941172 19679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:39 dist-test-slave-1t6p krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:39.947527 19679 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:17:39.949040 19679 webserver.cc:460] Webserver started at http://127.0.97.62:33239/ using document root <none> and password file <none>
I20260927 05:17:39.950048 19679 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:39.950187 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:39.950505 19679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:39.953403 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "d423e82aeb4543c9b3b501d1f8593769"
format_stamp: "Formatted at 2026-09-27 05:17:39 on dist-test-slave-1t6p"
server_key: "29fa36d25b827d5bdb9118ce374f64f9"
server_key_iv: "2a799044e87860f9dae799230b163e79"
server_key_version: "encryptionkey@0"
I20260927 05:17:39.954136 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "d423e82aeb4543c9b3b501d1f8593769"
format_stamp: "Formatted at 2026-09-27 05:17:39 on dist-test-slave-1t6p"
server_key: "29fa36d25b827d5bdb9118ce374f64f9"
server_key_iv: "2a799044e87860f9dae799230b163e79"
server_key_version: "encryptionkey@0"
I20260927 05:17:39.958964 19679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 05:17:39.962954 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:39.964514 19679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:17:39.964686 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "d423e82aeb4543c9b3b501d1f8593769"
format_stamp: "Formatted at 2026-09-27 05:17:39 on dist-test-slave-1t6p"
server_key: "29fa36d25b827d5bdb9118ce374f64f9"
server_key_iv: "2a799044e87860f9dae799230b163e79"
server_key_version: "encryptionkey@0"
I20260927 05:17:39.964895 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:39.977077 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:39.981204 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:39.981475 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:40.000023 19679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34095
I20260927 05:17:40.000058 19745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34095 every 8 connection(s)
I20260927 05:17:40.001389 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:40.004930 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:40.008016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19679
I20260927 05:17:40.008199 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:40.008730 388 external_mini_cluster.cc:1428] Setting key 03d01cf871a85771f1bb32e41d654ed3
I20260927 05:17:40.017165 19746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: Bootstrap starting.
I20260927 05:17:40.020838 19746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:40.022113 19746 log.cc:824] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:17:40 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:40.025118 19746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: No bootstrap required, opened a new log
I20260927 05:17:40.031832 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:40.010646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.97.62:34095)
0927 05:17:40.011447 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:40.011458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:40.011497 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:17:40.013119 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:40.013140 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:40.013560 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:40.013876 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:40.015667 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.016952 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.018764 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.019503 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.019630 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:40.024389 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:40.024401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:40.024418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:40.024459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:40.027784 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:40.028648 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:40.028655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:40.028660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:40.028843 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:40.029250 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:40.029254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:40.029256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:40.029747 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:40.029951 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:40.030407 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 05:17:40.030764 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":369,"threads_started":1}
I20260927 05:17:40.034390 19746 raft_consensus.cc:348] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } }
I20260927 05:17:40.034742 19746 raft_consensus.cc:374] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:40.034861 19746 raft_consensus.cc:729] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d423e82aeb4543c9b3b501d1f8593769, State: Initialized, Role: FOLLOWER
I20260927 05:17:40.035446 19746 consensus_queue.cc:260] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } }
I20260927 05:17:40.035702 19746 raft_consensus.cc:388] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:40.035851 19746 raft_consensus.cc:482] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:40.036026 19746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:40.038065 19746 raft_consensus.cc:504] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } }
I20260927 05:17:40.038570 19746 leader_election.cc:302] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d423e82aeb4543c9b3b501d1f8593769; no voters:
I20260927 05:17:40.039191 19746 leader_election.cc:288] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:40.039276 19751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:40.040261 19751 raft_consensus.cc:686] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [term 1 LEADER]: Becoming Leader. State: Replica: d423e82aeb4543c9b3b501d1f8593769, State: Running, Role: LEADER
I20260927 05:17:40.040838 19751 consensus_queue.cc:237] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } }
I20260927 05:17:40.041294 19746 sys_catalog.cc:564] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:40.044049 19752 sys_catalog.cc:455] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d423e82aeb4543c9b3b501d1f8593769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } } }
I20260927 05:17:40.044479 19752 sys_catalog.cc:458] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:40.044948 19753 sys_catalog.cc:455] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d423e82aeb4543c9b3b501d1f8593769. Latest consensus state: current_term: 1 leader_uuid: "d423e82aeb4543c9b3b501d1f8593769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d423e82aeb4543c9b3b501d1f8593769" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34095 } } }
I20260927 05:17:40.045332 19753 sys_catalog.cc:458] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:40.047554 19760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:40.051985 19760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:40.059674 19760 catalog_manager.cc:1347] Generated new cluster ID: be35754769e0477481a799c94d0d9ed3
I20260927 05:17:40.059778 19760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:40.073601 19760 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:40.075390 19760 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:40.085410 19760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: Generated new TSK 0
I20260927 05:17:40.086329 19760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:17:40.148367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:40.463593 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:40.463833 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:40.463989 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:40.470885 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:40.470971 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:40.471189 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:40.480084 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:40.481223 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:40.482796 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:40.497239 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:40.497279 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:40.497198 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:40.497915 19772 server_base.cc:1034] running on GCE node
I20260927 05:17:40.498479 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:40.499514 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:40.500747 19772 hybrid_clock.cc:648] HybridClock initialized: now 1790486260500710 us; error 37 us; skew 500 ppm
I20260927 05:17:40.501051 19772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:40 dist-test-slave-1t6p krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:40.504083 19772 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:17:40.505470 19772 webserver.cc:460] Webserver started at http://127.0.97.1:36933/ using document root <none> and password file <none>
I20260927 05:17:40.506163 19772 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:40.506304 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:40.506597 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:40.509410 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "b47a171810fa4c4caaf259aa23fbd2b5"
format_stamp: "Formatted at 2026-09-27 05:17:40 on dist-test-slave-1t6p"
server_key: "3f730979b5bfb7811db73adb5eafd5f9"
server_key_iv: "c0a504b764a799c454187343ce50d0bc"
server_key_version: "encryptionkey@0"
I20260927 05:17:40.510172 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b47a171810fa4c4caaf259aa23fbd2b5"
format_stamp: "Formatted at 2026-09-27 05:17:40 on dist-test-slave-1t6p"
server_key: "3f730979b5bfb7811db73adb5eafd5f9"
server_key_iv: "c0a504b764a799c454187343ce50d0bc"
server_key_version: "encryptionkey@0"
I20260927 05:17:40.514905 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 05:17:40.518066 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:40.519325 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 05:17:40.519487 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "b47a171810fa4c4caaf259aa23fbd2b5"
format_stamp: "Formatted at 2026-09-27 05:17:40 on dist-test-slave-1t6p"
server_key: "3f730979b5bfb7811db73adb5eafd5f9"
server_key_iv: "c0a504b764a799c454187343ce50d0bc"
server_key_version: "encryptionkey@0"
I20260927 05:17:40.519682 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:40.543529 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:40.547605 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:40.547895 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:40.548815 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:40.550053 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:40.550154 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:40.550294 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:40.550364 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:40.583180 19772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33951
I20260927 05:17:40.583262 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33951 every 8 connection(s)
I20260927 05:17:40.584390 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:40.590093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19772
I20260927 05:17:40.590273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:40.590814 388 external_mini_cluster.cc:1428] Setting key 155923539f959dab379d10f17485ffd3
Sep 27 05:17:40 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:40.607270 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:40.586994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56679 (local address 127.0.97.62:34095)
0927 05:17:40.587447 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:40.587453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:40.588529 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 05:17:40.590725 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:40.590730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:40.590803 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:40.590953 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:40.592770 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.593498 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.596118 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.596785 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.596916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:40.601416 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:40.601430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:40.601433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:40.601472 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:40.604235 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:40.604960 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:40.604965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:40.604967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:40.605081 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:40.605495 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:40.605500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:40.605503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:40.605784 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:40.605959 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:40.606650 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 05:17:40.606891 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260927 05:17:40.608197 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:40.587561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34095 (local address 127.0.97.1:56679)
0927 05:17:40.588370 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:40.588415 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:40.590381 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:40.591223 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:40.591235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:40.591932 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:40.592569 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:40.592609 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.593690 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.593699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:40.595954 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:40.595971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:40.597020 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:40.597029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:40.597360 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:40.598131 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:40.598163 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:40.601160 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:40.604380 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:40.604388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:40.604404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:40.604825 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:40.605196 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:40.605201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:40.605204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:40.605376 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:40.605925 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:40.605934 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:40.606250 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 05:17:40.606541 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 05:17:40.607060 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":337,"threads_started":1}
I20260927 05:17:40.609810 19900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34095
I20260927 05:17:40.610114 19900 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:40.610814 19900 heartbeater.cc:511] Master 127.0.97.62:34095 requested a full tablet report, sending...
I20260927 05:17:40.612903 19711 ts_manager.cc:194] Registered new tserver with Master: b47a171810fa4c4caaf259aa23fbd2b5 (127.0.97.1:33951)
I20260927 05:17:40.614315 19711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56679
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:17:40.646484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:40.961439 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:40.961694 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:40.961846 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:40.970671 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:40.970767 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:40.970948 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:40.980129 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:40.981271 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:40.982492 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:40.995947 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:40.995954 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:40.996075 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:40.996857 19906 server_base.cc:1034] running on GCE node
I20260927 05:17:40.997439 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:40.998306 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:40.999542 19906 hybrid_clock.cc:648] HybridClock initialized: now 1790486260999497 us; error 22 us; skew 500 ppm
I20260927 05:17:40.999883 19906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:40 dist-test-slave-1t6p krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:41.002971 19906 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:17:41.004424 19906 webserver.cc:460] Webserver started at http://127.0.97.2:37007/ using document root <none> and password file <none>
I20260927 05:17:41.005069 19906 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:41.005210 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:41.005502 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:41.008276 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "b493a48a4b8e7131710c879b2a6f4531"
server_key_iv: "924e7cf2ab59cd57f7323a8cb15db483"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.009021 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "b493a48a4b8e7131710c879b2a6f4531"
server_key_iv: "924e7cf2ab59cd57f7323a8cb15db483"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.013859 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 05:17:41.017031 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.018278 19906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:17:41.018438 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "b493a48a4b8e7131710c879b2a6f4531"
server_key_iv: "924e7cf2ab59cd57f7323a8cb15db483"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.018616 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:41.036098 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:41.040256 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:41.040518 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:41.041399 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:41.042695 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:41.042791 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.042937 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:41.043005 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.075207 19906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42915
I20260927 05:17:41.075253 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42915 every 8 connection(s)
I20260927 05:17:41.076370 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:41.077557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 19906
I20260927 05:17:41.077795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:41.078329 388 external_mini_cluster.cc:1428] Setting key 9eb98ea061a45b1b5b26adb100456f1b
Sep 27 05:17:41 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:41.097162 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.079231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40393 (local address 127.0.97.62:34095)
0927 05:17:41.079415 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.079422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.080653 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 05:17:41.082441 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.082446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.082533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.082642 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:41.084101 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.084818 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.086605 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.087073 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.087206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.090649 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:41.090655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:41.090659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:41.090691 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:41.093993 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.094664 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.094672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.094678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.094845 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.095304 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.095312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.095318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.095691 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:41.095839 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.096449 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.096703 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:17:41.097733 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.079683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34095 (local address 127.0.97.2:40393)
0927 05:17:41.080512 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.080553 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.082182 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.082867 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.082875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.083425 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:41.083939 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.083955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.084969 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.084974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.086467 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.086478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.087286 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.087292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.087478 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.088135 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:41.088156 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:41.090436 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:41.094128 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:41.094135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:41.094145 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:41.094542 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:41.094949 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:41.094953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:41.094955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:41.095194 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:41.095808 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:41.095814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:41.096089 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.096302 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.096675 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":310,"threads_started":1}
I20260927 05:17:41.099277 20034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34095
I20260927 05:17:41.099625 20034 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:41.100610 20034 heartbeater.cc:511] Master 127.0.97.62:34095 requested a full tablet report, sending...
I20260927 05:17:41.102226 19711 ts_manager.cc:194] Registered new tserver with Master: f4a7e64bea3a4e9cb2e94a2530e79a23 (127.0.97.2:42915)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:17:41.103116 19711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:17:41.133805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:41.444799 20039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:41.445039 20039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:41.445231 20039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:41.452054 20039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:41.452174 20039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:41.452358 20039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:41.461409 20039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:41.462929 20039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:41.464179 20039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:41.476886 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:41.476967 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:41.476929 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:41.477893 20039 server_base.cc:1034] running on GCE node
I20260927 05:17:41.478454 20039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:41.479384 20039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:41.480600 20039 hybrid_clock.cc:648] HybridClock initialized: now 1790486261480564 us; error 58 us; skew 500 ppm
I20260927 05:17:41.480894 20039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:41 dist-test-slave-1t6p krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:41.483889 20039 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:17:41.485286 20039 webserver.cc:460] Webserver started at http://127.0.97.3:40543/ using document root <none> and password file <none>
I20260927 05:17:41.485905 20039 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:41.485999 20039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:41.486230 20039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:41.488943 20039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "b65e4a22416a46e4b1e84e9e48db9d94"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "3ffd0c7f9ec7b9f9d076fdfe5cd1a30c"
server_key_iv: "722e23484e8ea07358635f59ae6b67b8"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.489667 20039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b65e4a22416a46e4b1e84e9e48db9d94"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "3ffd0c7f9ec7b9f9d076fdfe5cd1a30c"
server_key_iv: "722e23484e8ea07358635f59ae6b67b8"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.494438 20039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 05:17:41.497722 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.498926 20039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:17:41.499148 20039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "b65e4a22416a46e4b1e84e9e48db9d94"
format_stamp: "Formatted at 2026-09-27 05:17:41 on dist-test-slave-1t6p"
server_key: "3ffd0c7f9ec7b9f9d076fdfe5cd1a30c"
server_key_iv: "722e23484e8ea07358635f59ae6b67b8"
server_key_version: "encryptionkey@0"
I20260927 05:17:41.499328 20039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:41.514077 20039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:41.517838 20039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:41.518110 20039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:41.518992 20039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:41.520292 20039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:41.520373 20039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.520546 20039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:41.520597 20039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:41.554468 20039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39017
I20260927 05:17:41.554508 20166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39017 every 8 connection(s)
I20260927 05:17:41.555738 20039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:41.564544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20039
I20260927 05:17:41.564718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:41.565146 388 external_mini_cluster.cc:1428] Setting key 15d72655b4ed93d3fa5cd7d476fb8926
Sep 27 05:17:41 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:17:41.574331 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.558131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37607 (local address 127.0.97.62:34095)
0927 05:17:41.558315 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.558320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.559440 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 05:17:41.561177 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.561183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.561277 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.561385 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:41.562760 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.563615 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.565250 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.565789 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.565915 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.569388 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:41.569393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:41.569396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:41.569426 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:41.571927 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.572559 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.572563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.572565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.572683 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.573021 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.573025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.573026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.573271 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:41.573397 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.573928 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.574081 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:17:41.575273 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.558585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34095 (local address 127.0.97.3:37607)
0927 05:17:41.559297 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.559334 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.560927 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.561599 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.561608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.562081 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:41.562597 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.562613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.563775 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.563779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.565068 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.565080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.566006 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.566016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.566283 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.566967 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:17:41.566988 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:41.569212 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:41.572049 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:41.572055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:41.572066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:41.572465 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:41.572774 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:17:41.572777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:17:41.572780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:17:41.572935 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:17:41.573381 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:41.573386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:41.573643 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.573842 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.574157 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":269,"threads_started":1}
I20260927 05:17:41.576447 20167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34095
I20260927 05:17:41.576683 20167 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:41.577275 20167 heartbeater.cc:511] Master 127.0.97.62:34095 requested a full tablet report, sending...
I20260927 05:17:41.578703 19711 ts_manager.cc:194] Registered new tserver with Master: b65e4a22416a46e4b1e84e9e48db9d94 (127.0.97.3:39017)
I20260927 05:17:41.579762 19711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37607
I20260927 05:17:41.579972 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:41.596299 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.583970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.97.62:34095)
0927 05:17:41.584126 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.584131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.584636 (+ 505us) server_negotiation.cc:407] Connection header received
0927 05:17:41.584754 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.584760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.584827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.584922 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:41.586187 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.587397 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.588649 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.589032 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.589135 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.591399 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:41.591404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:41.591407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:41.591435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:41.593948 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.594515 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.594519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.594521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.594640 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.594956 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.594959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.594961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.595296 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:41.595516 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.595843 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.596078 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:17:41.599920 19711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:41.606675 19711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:41.617115 19900 heartbeater.cc:503] Master 127.0.97.62:34095 was elected leader, sending a full tablet report...
I20260927 05:17:41.640166 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.618483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39017 (local address 127.0.0.1:42934)
0927 05:17:41.628340 (+ 9857us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.628361 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.628476 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.633386 (+ 4910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.633394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.633444 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:41.633866 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.633880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635841 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.635845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.637486 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.637498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.639096 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.639105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.639322 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.639372 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.639503 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.639670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9742,"mutex_wait_us":37,"spinlock_wait_cycles":15488,"thread_start_us":9625,"threads_started":1}
I20260927 05:17:41.640945 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.618226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42915 (local address 127.0.0.1:49342)
0927 05:17:41.628239 (+ 10013us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.628297 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.628476 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.632731 (+ 4255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.632756 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.632787 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:41.633227 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.633259 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635472 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.635476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.638157 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.638171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.639976 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.639986 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.640203 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.640236 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.640320 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.640466 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9715,"mutex_wait_us":195,"spinlock_wait_cycles":6784,"thread_start_us":1148,"threads_started":1}
I20260927 05:17:41.640992 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.620254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33951 (local address 127.0.0.1:51496)
0927 05:17:41.628952 (+ 8698us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.628977 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.629095 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.632933 (+ 3838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.632938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.632979 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:41.633480 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.633495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635410 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.635417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.638158 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.638171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.640123 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.640130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.640287 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.640322 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.640390 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.640488 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8531,"mutex_wait_us":136,"spinlock_wait_cycles":896,"thread_start_us":7578,"threads_started":1}
I20260927 05:17:41.641441 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.620818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.97.1:33951)
0927 05:17:41.632381 (+ 11563us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.632392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.632425 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:17:41.632531 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.632537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.632750 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.632957 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:41.633683 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635288 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.638679 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.639876 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.640639 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.640868 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.640950 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.641108 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11358,"thread_start_us":11175,"threads_started":1}
I20260927 05:17:41.641610 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.619032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.97.3:39017)
0927 05:17:41.632808 (+ 13776us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.632817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.632850 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:17:41.632943 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.632951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.633147 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.634046 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:41.634304 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635697 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.637640 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.638718 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.640658 (+ 1940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.640887 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.640956 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.641291 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13603,"thread_start_us":13446,"threads_started":1}
I20260927 05:17:41.642416 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.618850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.0.97.2:42915)
0927 05:17:41.632061 (+ 13211us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.632071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.632097 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:17:41.632217 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.632227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.632487 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.633456 (+ 969us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:41.633667 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.635326 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.638321 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.639754 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.639882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.640077 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.641186 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.641343 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13024,"thread_start_us":13351,"threads_started":1}
I20260927 05:17:41.645157 19835 tablet_service.cc:1467] Processing CreateTablet for tablet 31ee0bb461574efeb5b132c9768e8559 (DEFAULT_TABLE table=test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:41.645157 20102 tablet_service.cc:1467] Processing CreateTablet for tablet 31ee0bb461574efeb5b132c9768e8559 (DEFAULT_TABLE table=test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:41.646657 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ee0bb461574efeb5b132c9768e8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:41.646658 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ee0bb461574efeb5b132c9768e8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:41.651463 19969 tablet_service.cc:1467] Processing CreateTablet for tablet 31ee0bb461574efeb5b132c9768e8559 (DEFAULT_TABLE table=test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:41.652503 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ee0bb461574efeb5b132c9768e8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:41.668411 20185 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Bootstrap starting.
I20260927 05:17:41.669456 20186 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Bootstrap starting.
I20260927 05:17:41.670377 20187 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Bootstrap starting.
I20260927 05:17:41.673092 20186 tablet_bootstrap.cc:654] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:41.673501 20185 tablet_bootstrap.cc:654] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:41.673811 20187 tablet_bootstrap.cc:654] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:41.674196 20186 log.cc:824] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:41.674701 20185 log.cc:824] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:41.674912 20187 log.cc:824] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:41.676746 20186 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: No bootstrap required, opened a new log
I20260927 05:17:41.676997 20186 ts_tablet_manager.cc:1397] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:17:41.677693 20187 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: No bootstrap required, opened a new log
I20260927 05:17:41.678194 20187 ts_tablet_manager.cc:1397] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 05:17:41.679034 20185 tablet_bootstrap.cc:492] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: No bootstrap required, opened a new log
I20260927 05:17:41.679275 20185 ts_tablet_manager.cc:1397] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 05:17:41.684669 20186 raft_consensus.cc:348] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.685007 20186 raft_consensus.cc:374] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:41.685130 20186 raft_consensus.cc:729] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65e4a22416a46e4b1e84e9e48db9d94, State: Initialized, Role: FOLLOWER
I20260927 05:17:41.685721 20186 consensus_queue.cc:260] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.686694 20167 heartbeater.cc:503] Master 127.0.97.62:34095 was elected leader, sending a full tablet report...
I20260927 05:17:41.688504 20186 ts_tablet_manager.cc:1428] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 05:17:41.688656 20187 raft_consensus.cc:348] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.689049 20187 raft_consensus.cc:374] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:41.689149 20187 raft_consensus.cc:729] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a7e64bea3a4e9cb2e94a2530e79a23, State: Initialized, Role: FOLLOWER
I20260927 05:17:41.689575 20185 raft_consensus.cc:348] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.689752 20187 consensus_queue.cc:260] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.690146 20185 raft_consensus.cc:374] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:41.690320 20185 raft_consensus.cc:729] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47a171810fa4c4caaf259aa23fbd2b5, State: Initialized, Role: FOLLOWER
I20260927 05:17:41.691102 20185 consensus_queue.cc:260] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.693149 20185 ts_tablet_manager.cc:1428] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 05:17:41.694077 20034 heartbeater.cc:503] Master 127.0.97.62:34095 was elected leader, sending a full tablet report...
I20260927 05:17:41.694680 20187 ts_tablet_manager.cc:1428] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260927 05:17:41.807420 20168 tablet.cc:2369] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:17:41.828348 20035 tablet.cc:2369] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:41.828558 20191 raft_consensus.cc:482] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:41.828881 20191 raft_consensus.cc:504] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.830545 20191 leader_election.cc:288] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4a7e64bea3a4e9cb2e94a2530e79a23 (127.0.97.2:42915), b47a171810fa4c4caaf259aa23fbd2b5 (127.0.97.1:33951)
I20260927 05:17:41.835886 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.830722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42915 (local address 127.0.97.3:36439)
0927 05:17:41.830910 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.830930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.831078 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.831562 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.831567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.831603 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:41.831781 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.831788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.832778 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.832789 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.834046 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.834057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.835149 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.835155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.835332 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.835354 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.835459 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.835588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:17:41.835886 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.830929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36439 (local address 127.0.97.2:42915)
0927 05:17:41.831142 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.831151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.831172 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:17:41.831246 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.831251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.831324 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.831425 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:41.831925 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.832621 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.834219 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.834965 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.835132 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.835220 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.835484 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.835613 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 05:17:41.836823 19901 tablet.cc:2369] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:41.837018 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.831099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33951 (local address 127.0.97.3:54419)
0927 05:17:41.831769 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:41.831783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:41.831916 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:41.832263 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:41.832267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:41.832289 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:41.832587 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.832595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.833561 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.833566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.834957 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:41.834968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.836209 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.836217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:41.836402 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.836441 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:17:41.836530 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 05:17:41.836682 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
I20260927 05:17:41.837066 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.831264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54419 (local address 127.0.97.1:33951)
0927 05:17:41.831454 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.831459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.831874 (+ 415us) server_negotiation.cc:407] Connection header received
0927 05:17:41.832065 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.832071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.832130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.832250 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:41.832730 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.833405 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.835147 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.835929 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.836103 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.836187 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.836585 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.836760 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:17:41.837008 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ee0bb461574efeb5b132c9768e8559" candidate_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" is_pre_election: true
I20260927 05:17:41.837579 19989 raft_consensus.cc:2454] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65e4a22416a46e4b1e84e9e48db9d94 in term 0.
I20260927 05:17:41.838310 20058 leader_election.cc:302] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65e4a22416a46e4b1e84e9e48db9d94, f4a7e64bea3a4e9cb2e94a2530e79a23; no voters:
I20260927 05:17:41.838231 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ee0bb461574efeb5b132c9768e8559" candidate_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" is_pre_election: true
I20260927 05:17:41.838729 20191 raft_consensus.cc:2781] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:41.838738 19855 raft_consensus.cc:2454] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65e4a22416a46e4b1e84e9e48db9d94 in term 0.
I20260927 05:17:41.838886 20191 raft_consensus.cc:482] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:41.839035 20191 raft_consensus.cc:3037] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:41.841451 20191 raft_consensus.cc:504] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.842017 20191 leader_election.cc:288] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [CANDIDATE]: Term 1 election: Requested vote from peers f4a7e64bea3a4e9cb2e94a2530e79a23 (127.0.97.2:42915), b47a171810fa4c4caaf259aa23fbd2b5 (127.0.97.1:33951)
I20260927 05:17:41.842511 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ee0bb461574efeb5b132c9768e8559" candidate_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47a171810fa4c4caaf259aa23fbd2b5"
I20260927 05:17:41.842532 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ee0bb461574efeb5b132c9768e8559" candidate_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23"
I20260927 05:17:41.842697 19855 raft_consensus.cc:3037] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:41.842725 19989 raft_consensus.cc:3037] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:41.844939 19989 raft_consensus.cc:2454] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65e4a22416a46e4b1e84e9e48db9d94 in term 1.
I20260927 05:17:41.845180 19855 raft_consensus.cc:2454] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65e4a22416a46e4b1e84e9e48db9d94 in term 1.
I20260927 05:17:41.845402 20058 leader_election.cc:302] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65e4a22416a46e4b1e84e9e48db9d94, f4a7e64bea3a4e9cb2e94a2530e79a23; no voters:
I20260927 05:17:41.845667 20191 raft_consensus.cc:2781] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:41.846091 20191 raft_consensus.cc:686] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 1 LEADER]: Becoming Leader. State: Replica: b65e4a22416a46e4b1e84e9e48db9d94, State: Running, Role: LEADER
I20260927 05:17:41.846575 20191 consensus_queue.cc:237] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } }
I20260927 05:17:41.850687 19711 catalog_manager.cc:5495] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65e4a22416a46e4b1e84e9e48db9d94 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65e4a22416a46e4b1e84e9e48db9d94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 } health_report { overall_health: UNKNOWN } } }
I20260927 05:17:41.873729 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.868293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.97.3:39017)
0927 05:17:41.868484 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.868491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.868535 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:17:41.868784 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.868787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.868869 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.869018 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:41.869434 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.870141 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.871727 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.872114 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.872256 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.872731 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:41.873191 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:41.873343 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.873408 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.873482 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:17:41.881513 19989 raft_consensus.cc:1261] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 1 FOLLOWER]: Refusing update from remote peer b65e4a22416a46e4b1e84e9e48db9d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:41.881531 19855 raft_consensus.cc:1261] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 1 FOLLOWER]: Refusing update from remote peer b65e4a22416a46e4b1e84e9e48db9d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:41.882396 20195 consensus_queue.cc:1035] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4a7e64bea3a4e9cb2e94a2530e79a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:41.882781 20191 consensus_queue.cc:1035] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47a171810fa4c4caaf259aa23fbd2b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:41.893554 20198 mvcc.cc:205] Tried to move back new op lower bound from 7333831728657203200 to 7333831728527331328. Current Snapshot: MvccSnapshot[applied={T|T < 7333831728657203200}]
I20260927 05:17:41.911386 19710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:41.911672 19710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:41.915284 19710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d423e82aeb4543c9b3b501d1f8593769: Sending DeleteTablet for 3 replicas of tablet 31ee0bb461574efeb5b132c9768e8559
I20260927 05:17:41.916463 20102 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ee0bb461574efeb5b132c9768e8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42934
I20260927 05:17:41.916544 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ee0bb461574efeb5b132c9768e8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49342
I20260927 05:17:41.917404 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ee0bb461574efeb5b132c9768e8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260927 05:17:41.917747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19772
I20260927 05:17:41.923997 20208 tablet_replica.cc:330] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: stopping tablet replica
I20260927 05:17:41.924343 20208 raft_consensus.cc:2229] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:41.924832 20208 raft_consensus.cc:2258] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:41.926791 20208 ts_tablet_manager.cc:1905] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:41.934078 20208 ts_tablet_manager.cc:1918] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:41.934288 20208 log.cc:1196] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/31ee0bb461574efeb5b132c9768e8559
I20260927 05:17:41.934907 20208 ts_tablet_manager.cc:1939] T 31ee0bb461574efeb5b132c9768e8559 P b65e4a22416a46e4b1e84e9e48db9d94: Deleting consensus metadata
I20260927 05:17:41.936630 19697 catalog_manager.cc:4841] TS b65e4a22416a46e4b1e84e9e48db9d94 (127.0.97.3:39017): tablet 31ee0bb461574efeb5b132c9768e8559 (table test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]) successfully deleted
I20260927 05:17:41.937441 20209 tablet_replica.cc:330] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: stopping tablet replica
I20260927 05:17:41.937934 20210 tablet_replica.cc:330] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: stopping tablet replica
I20260927 05:17:41.937906 20209 raft_consensus.cc:2229] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:41.938314 20210 raft_consensus.cc:2229] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:41.938416 20209 raft_consensus.cc:2258] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:41.938781 20210 raft_consensus.cc:2258] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:41.941244 20209 ts_tablet_manager.cc:1905] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:41.941543 20210 ts_tablet_manager.cc:1905] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:17:41 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:17:41.948339 20209 ts_tablet_manager.cc:1918] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:41.948567 20209 log.cc:1196] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/31ee0bb461574efeb5b132c9768e8559
I20260927 05:17:41.949146 20209 ts_tablet_manager.cc:1939] T 31ee0bb461574efeb5b132c9768e8559 P f4a7e64bea3a4e9cb2e94a2530e79a23: Deleting consensus metadata
I20260927 05:17:41.949163 20210 ts_tablet_manager.cc:1918] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:41.949375 20210 log.cc:1196] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/31ee0bb461574efeb5b132c9768e8559
I20260927 05:17:41.949941 20210 ts_tablet_manager.cc:1939] T 31ee0bb461574efeb5b132c9768e8559 P b47a171810fa4c4caaf259aa23fbd2b5: Deleting consensus metadata
I20260927 05:17:41.950922 19698 catalog_manager.cc:4841] TS f4a7e64bea3a4e9cb2e94a2530e79a23 (127.0.97.2:42915): tablet 31ee0bb461574efeb5b132c9768e8559 (table test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]) successfully deleted
I20260927 05:17:41.952013 19698 catalog_manager.cc:4841] TS b47a171810fa4c4caaf259aa23fbd2b5 (127.0.97.1:33951): tablet 31ee0bb461574efeb5b132c9768e8559 (table test-table [id=10186b9b0f7f4f7f9dbf07b17982ea1a]) successfully deleted
I20260927 05:17:41.953146 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:41.919262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.97.1:33951)
0927 05:17:41.919470 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:41.919479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:41.930074 (+ 10595us) server_negotiation.cc:407] Connection header received
0927 05:17:41.930537 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:41.930546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:41.930681 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:41.930766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:41.932888 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.934287 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.938927 (+ 4640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:41.939637 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:41.939775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:41.945454 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:41.945468 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:41.945484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:41.945554 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:41.949743 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.950620 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.950628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.950631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.950791 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:41.951275 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:41.951281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:41.951285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:41.951797 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:41.952090 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:41.952625 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 05:17:41.952806 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:41.953823 19895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:42.045389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19772
I20260927 05:17:42.063373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19906
Sep 27 05:17:42 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:17:42.081457 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:42.064287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.0.97.2:42915)
0927 05:17:42.064516 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:42.064524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:42.064547 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:17:42.064619 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:42.064625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:42.064716 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:42.064845 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:42.066160 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:42.067554 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:42.068981 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:42.069708 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:42.069844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:42.073881 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:42.073899 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:42.073920 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:42.073966 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:42.078288 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:42.079093 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:42.079105 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:42.079109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:42.079280 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:42.079742 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:42.079747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:42.079751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:42.080228 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:42.080480 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:42.080882 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 05:17:42.081098 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:17:42.082163 20029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:42.223349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19906
I20260927 05:17:42.248396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20039
Sep 27 05:17:42 dist-test-slave-1t6p krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:17:42.267633 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:42.249339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.97.3:39017)
0927 05:17:42.249507 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:42.249512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:42.249609 (+ 97us) server_negotiation.cc:407] Connection header received
0927 05:17:42.249823 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:42.249827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:42.249887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:42.249976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:42.252054 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:42.252904 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:42.254996 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:42.255520 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:42.255618 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:42.261391 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:42.261401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:42.261415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:42.261454 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:42.264824 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:42.265481 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:42.265487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:42.265490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:42.265661 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:42.266049 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:42.266053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:42.266059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:42.266401 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:42.266736 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:42.266969 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 05:17:42.267178 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:42.268519 20162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:42.386579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20039
I20260927 05:17:42.405632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 19679
I20260927 05:17:42.406764 19741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:42.517482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 19679
2026-09-27T05:17:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4261 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T05:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:42Z Disabled control of system clock
I20260927 05:17:42.589779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42595
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:42.905905 20219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:42.906167 20219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:42.906308 20219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:42.913431 20219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:42.913523 20219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:42.913606 20219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:42.913715 20219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:42.913802 20219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:42.922636 20219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:42.923859 20219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:42.925562 20219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:42.936859 20224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:42.936929 20227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:42.936895 20225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:42.937656 20219 server_base.cc:1034] running on GCE node
I20260927 05:17:42.938416 20219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:42.939705 20219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:42.940896 20219 hybrid_clock.cc:648] HybridClock initialized: now 1790486262940856 us; error 32 us; skew 500 ppm
I20260927 05:17:42.941171 20219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:42.943387 20219 webserver.cc:460] Webserver started at http://127.0.97.62:38547/ using document root <none> and password file <none>
I20260927 05:17:42.943958 20219 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:42.944037 20219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:42.944279 20219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:42.946914 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "53dd509e9a1747a99076963ffa2cf488"
format_stamp: "Formatted at 2026-09-27 05:17:42 on dist-test-slave-1t6p"
server_key: "972ed3c4543f59a732e3d04199abf70c"
server_key_iv: "4f156f2446aeea3a4b8834cfb980a644"
server_key_version: "encryptionkey@0"
I20260927 05:17:42.947679 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "53dd509e9a1747a99076963ffa2cf488"
format_stamp: "Formatted at 2026-09-27 05:17:42 on dist-test-slave-1t6p"
server_key: "972ed3c4543f59a732e3d04199abf70c"
server_key_iv: "4f156f2446aeea3a4b8834cfb980a644"
server_key_version: "encryptionkey@0"
I20260927 05:17:42.952397 20219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:17:42.955493 20233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:42.956750 20219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:17:42.956888 20219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "53dd509e9a1747a99076963ffa2cf488"
format_stamp: "Formatted at 2026-09-27 05:17:42 on dist-test-slave-1t6p"
server_key: "972ed3c4543f59a732e3d04199abf70c"
server_key_iv: "4f156f2446aeea3a4b8834cfb980a644"
server_key_version: "encryptionkey@0"
I20260927 05:17:42.957057 20219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:42.987782 20219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:42.988626 20219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:42.988878 20219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:43.006814 20219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42595
I20260927 05:17:43.006848 20284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42595 every 8 connection(s)
I20260927 05:17:43.008176 20219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:43.009258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20219
I20260927 05:17:43.009452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:43.009915 388 external_mini_cluster.cc:1428] Setting key bd04f9ee7e15738d18c9fa6bb381dd26
I20260927 05:17:43.012805 20285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:43.022951 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:43.011951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.0.97.62:42595)
0927 05:17:43.012958 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:43.012977 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:43.013019 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:17:43.014605 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:43.014626 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:43.015143 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:43.015539 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:43.016096 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.017383 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.018707 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.019438 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.019627 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:43.020781 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:43.020789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:43.020805 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:43.021376 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:43.021447 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:43.021466 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:43.021595 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:43.021789 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 05:17:43.022071 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":349,"threads_started":1}
I20260927 05:17:43.023296 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: Bootstrap starting.
I20260927 05:17:43.026584 20285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:43.027532 20285 log.cc:824] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:43.030241 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: No bootstrap required, opened a new log
I20260927 05:17:43.037130 20285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } }
I20260927 05:17:43.037448 20285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:43.037521 20285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dd509e9a1747a99076963ffa2cf488, State: Initialized, Role: FOLLOWER
I20260927 05:17:43.038012 20285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } }
I20260927 05:17:43.038199 20285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:43.038300 20285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:43.038422 20285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:43.040477 20285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } }
I20260927 05:17:43.040894 20285 leader_election.cc:302] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dd509e9a1747a99076963ffa2cf488; no voters:
I20260927 05:17:43.041388 20285 leader_election.cc:288] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:43.041536 20290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:43.042418 20290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [term 1 LEADER]: Becoming Leader. State: Replica: 53dd509e9a1747a99076963ffa2cf488, State: Running, Role: LEADER
I20260927 05:17:43.042879 20290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } }
I20260927 05:17:43.043431 20285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:43.045589 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dd509e9a1747a99076963ffa2cf488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } } }
I20260927 05:17:43.046197 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:43.046020 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dd509e9a1747a99076963ffa2cf488. Latest consensus state: current_term: 1 leader_uuid: "53dd509e9a1747a99076963ffa2cf488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd509e9a1747a99076963ffa2cf488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42595 } } }
I20260927 05:17:43.046592 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:43.048722 20297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:43.053226 20297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:43.061977 20297 catalog_manager.cc:1347] Generated new cluster ID: c76d13abc01145d0ba2fe83247ba4667
I20260927 05:17:43.062081 20297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:43.070902 20297 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:43.072616 20297 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:43.096130 20297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: Generated new TSK 0
I20260927 05:17:43.097215 20297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:43.107206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:17:43.430194 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:43.430501 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:43.430657 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:43.437747 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:43.437872 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:43.438045 20309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:17:43.447248 20309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:43.448639 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:43.450527 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:43.467945 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:43.467993 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:43.468016 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:43.469045 20309 server_base.cc:1034] running on GCE node
I20260927 05:17:43.469650 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:43.470472 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:43.471668 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790486263471616 us; error 53 us; skew 500 ppm
I20260927 05:17:43.471980 20309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:43.474119 20309 webserver.cc:460] Webserver started at http://127.0.97.1:39827/ using document root <none> and password file <none>
I20260927 05:17:43.474759 20309 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:43.474877 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:43.475239 20309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:43.477950 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data/instance:
uuid: "8fe2e55e3f85446595333df4879a1022"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "11787a90b2ece5deabfea02ccb2ccc57"
server_key_iv: "e028771c625b00fd81a5ba8dcfa843df"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.478705 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8fe2e55e3f85446595333df4879a1022"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "11787a90b2ece5deabfea02ccb2ccc57"
server_key_iv: "e028771c625b00fd81a5ba8dcfa843df"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.483546 20309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:17:43.486812 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.488173 20309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:43.488345 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal
uuid: "8fe2e55e3f85446595333df4879a1022"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "11787a90b2ece5deabfea02ccb2ccc57"
server_key_iv: "e028771c625b00fd81a5ba8dcfa843df"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.488554 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:43.501726 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:43.502456 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:43.502718 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:43.503702 20309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:43.505079 20309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:43.505200 20309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.505339 20309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:43.505429 20309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.537896 20309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36691
I20260927 05:17:43.537978 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36691 every 8 connection(s)
I20260927 05:17:43.539252 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:17:43.540161 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20309
I20260927 05:17:43.540326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:17:43.540864 388 external_mini_cluster.cc:1428] Setting key 3b5250ba98c6cff481d48a06e106e67d
I20260927 05:17:43.543830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:43.553622 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:43.542036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53679 (local address 127.0.97.62:42595)
0927 05:17:43.543350 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:43.543356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:43.544331 (+ 975us) server_negotiation.cc:407] Connection header received
0927 05:17:43.546193 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:43.546202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:43.546291 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:43.546415 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:43.547487 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.548665 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.550492 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.551253 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.551407 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:43.552903 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:43.552907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:43.552911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:43.552929 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:43.552966 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:43.552978 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:43.553051 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:43.553318 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 05:17:43.553397 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":1093,"threads_started":1}
I20260927 05:17:43.555068 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:43.542837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42595 (local address 127.0.97.1:53679)
0927 05:17:43.544164 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:43.544212 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:43.545935 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:43.546604 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:43.546612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:43.546696 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:43.547316 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:43.547340 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.548840 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.548845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:43.550308 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:43.550320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.551464 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.551468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:43.551759 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:43.552533 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:43.552539 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:43.552599 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:43.552701 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:43.553079 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:43.553092 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:43.553103 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:17:43.553275 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 05:17:43.553672 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":565,"threads_started":1}
I20260927 05:17:43.556573 20436 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42595
I20260927 05:17:43.556864 20436 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:43.557548 20436 heartbeater.cc:511] Master 127.0.97.62:42595 requested a full tablet report, sending...
I20260927 05:17:43.559623 20250 ts_manager.cc:194] Registered new tserver with Master: 8fe2e55e3f85446595333df4879a1022 (127.0.97.1:36691)
I20260927 05:17:43.561218 20250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53679
W20260927 05:17:43.869513 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:43.869839 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:43.870003 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:43.876857 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:43.876981 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:43.877159 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:17:43.885768 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:43.887056 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:43.888823 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:43.901618 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:43.901669 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:43.901715 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:43.902630 20440 server_base.cc:1034] running on GCE node
I20260927 05:17:43.903287 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:43.904218 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:43.905439 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790486263905400 us; error 29 us; skew 500 ppm
I20260927 05:17:43.905814 20440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:43.908113 20440 webserver.cc:460] Webserver started at http://127.0.97.2:43449/ using document root <none> and password file <none>
I20260927 05:17:43.908797 20440 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:43.908881 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:43.909133 20440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:43.911863 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data/instance:
uuid: "b419574cc8664508ae5c0ea8e0110429"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "2d8ac213c8af2013a31dee76180afc7a"
server_key_iv: "b794748b93c06df8af9ed40a380d9c0c"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.912578 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b419574cc8664508ae5c0ea8e0110429"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "2d8ac213c8af2013a31dee76180afc7a"
server_key_iv: "b794748b93c06df8af9ed40a380d9c0c"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.917330 20440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:43.920703 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.921967 20440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:17:43.922137 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal
uuid: "b419574cc8664508ae5c0ea8e0110429"
format_stamp: "Formatted at 2026-09-27 05:17:43 on dist-test-slave-1t6p"
server_key: "2d8ac213c8af2013a31dee76180afc7a"
server_key_iv: "b794748b93c06df8af9ed40a380d9c0c"
server_key_version: "encryptionkey@0"
I20260927 05:17:43.922365 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:43.944726 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:43.945477 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:43.945772 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:43.946682 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:43.948200 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:43.948343 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.948475 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:43.948562 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:43.981988 20440 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36759
I20260927 05:17:43.982074 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36759 every 8 connection(s)
I20260927 05:17:43.983315 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:17:43.989730 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20440
I20260927 05:17:43.989924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:17:43.990350 388 external_mini_cluster.cc:1428] Setting key 07a0e839e2850a398937c45c3220d650
I20260927 05:17:43.993484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:17:43.996820 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:43.985866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37411 (local address 127.0.97.62:42595)
0927 05:17:43.986038 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:43.986042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:43.987512 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 05:17:43.989286 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:43.989293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:43.989388 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:43.989505 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:43.990707 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.991636 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.993632 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.994362 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.994504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:43.995895 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:43.995904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:43.995911 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:43.995935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:43.995985 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:43.996004 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:43.996098 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:43.996323 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 05:17:43.996430 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:17:43.997610 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:43.986430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42595 (local address 127.0.97.2:37411)
0927 05:17:43.987356 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:43.987400 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:43.989020 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:43.989712 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:43.989721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:43.989805 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:43.990531 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:43.990551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.991817 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.991822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:43.993468 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:43.993481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:43.994586 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:43.994591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:43.994798 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:43.995560 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:43.995563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:43.995602 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:43.995681 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:43.996105 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:43.996113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:43.996119 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:17:43.996277 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 05:17:43.996563 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":274,"threads_started":1}
I20260927 05:17:43.999109 20567 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42595
I20260927 05:17:43.999415 20567 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:44.000269 20567 heartbeater.cc:511] Master 127.0.97.62:42595 requested a full tablet report, sending...
I20260927 05:17:44.001799 20250 ts_manager.cc:194] Registered new tserver with Master: b419574cc8664508ae5c0ea8e0110429 (127.0.97.2:36759)
I20260927 05:17:44.002590 20250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37411
W20260927 05:17:44.312186 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:44.312441 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:44.312525 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:44.319416 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:44.319499 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:44.319598 20570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:17:44.328423 20570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:44.329495 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:44.331182 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:44.343622 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:44.343665 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:44.343605 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:44.348302 20570 server_base.cc:1034] running on GCE node
I20260927 05:17:44.348841 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:44.349738 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:44.350927 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790486264350896 us; error 28 us; skew 500 ppm
I20260927 05:17:44.351351 20570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:17:44.353389 20570 webserver.cc:460] Webserver started at http://127.0.97.3:34993/ using document root <none> and password file <none>
I20260927 05:17:44.354043 20570 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:44.354151 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:44.354478 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:44.357167 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data/instance:
uuid: "acc74eb3e64b432d8e794fb2469eceb4"
format_stamp: "Formatted at 2026-09-27 05:17:44 on dist-test-slave-1t6p"
server_key: "b1126f9e5a3d62e860d99b5107065992"
server_key_iv: "586e33208fa163ee0e5d6efc6e3320ef"
server_key_version: "encryptionkey@0"
I20260927 05:17:44.357921 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal/instance:
uuid: "acc74eb3e64b432d8e794fb2469eceb4"
format_stamp: "Formatted at 2026-09-27 05:17:44 on dist-test-slave-1t6p"
server_key: "b1126f9e5a3d62e860d99b5107065992"
server_key_iv: "586e33208fa163ee0e5d6efc6e3320ef"
server_key_version: "encryptionkey@0"
I20260927 05:17:44.362826 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:17:44.365998 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:44.367298 20570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:44.367456 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal
uuid: "acc74eb3e64b432d8e794fb2469eceb4"
format_stamp: "Formatted at 2026-09-27 05:17:44 on dist-test-slave-1t6p"
server_key: "b1126f9e5a3d62e860d99b5107065992"
server_key_iv: "586e33208fa163ee0e5d6efc6e3320ef"
server_key_version: "encryptionkey@0"
I20260927 05:17:44.367667 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:44.383862 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:44.384548 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:44.384816 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:44.385689 20570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:17:44.387022 20570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:17:44.387144 20570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:44.387274 20570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:17:44.387362 20570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:44.419929 20570 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45439
I20260927 05:17:44.420017 20696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45439 every 8 connection(s)
I20260927 05:17:44.421171 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:17:44.428814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20570
I20260927 05:17:44.428983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:17:44.429533 388 external_mini_cluster.cc:1428] Setting key 9b3845b4701748c24af3b17b2d2c73b8
I20260927 05:17:44.433432 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.423601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57443 (local address 127.0.97.62:42595)
0927 05:17:44.423810 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.423816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.425081 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 05:17:44.426962 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.426970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.427111 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.427196 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:44.428131 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.429179 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.430747 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.431245 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.431375 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.432639 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:44.432645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:44.432650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:44.432676 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:44.432707 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:44.432724 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:44.432820 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.433025 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.433116 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:17:44.434288 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.424126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42595 (local address 127.0.97.3:57443)
0927 05:17:44.424945 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.424984 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.426655 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.427390 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.427398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.427478 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:17:44.427995 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.428010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.429342 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.429345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.430594 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.430605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.431449 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.431456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.431659 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.432343 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:17:44.432345 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:17:44.432380 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:17:44.432449 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:17:44.432820 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:17:44.432831 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:17:44.432841 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.433012 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.433370 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":263,"threads_started":1}
I20260927 05:17:44.435410 20697 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42595
I20260927 05:17:44.435639 20697 heartbeater.cc:464] Registering TS with master...
I20260927 05:17:44.436288 20697 heartbeater.cc:511] Master 127.0.97.62:42595 requested a full tablet report, sending...
I20260927 05:17:44.437489 20250 ts_manager.cc:194] Registered new tserver with Master: acc74eb3e64b432d8e794fb2469eceb4 (127.0.97.3:45439)
I20260927 05:17:44.438211 20250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57443
I20260927 05:17:44.444085 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:17:44.453430 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.448025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.97.62:42595)
0927 05:17:44.448166 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.448169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.448664 (+ 495us) server_negotiation.cc:407] Connection header received
0927 05:17:44.448795 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.448798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.448877 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.448950 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:44.449371 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.450056 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.451374 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.451854 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.451945 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.452903 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:44.452907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:44.452910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:44.452925 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:44.452953 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:44.452964 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:44.453032 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.453148 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.453210 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:17:44.456862 20250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:17:44.463677 20250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:17:44.489892 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.474834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45439 (local address 127.0.0.1:47650)
0927 05:17:44.475720 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.475758 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.475918 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.477328 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.477340 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.477380 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:44.479072 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.479094 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.481504 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.481512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.486948 (+ 5436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.486964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.488966 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.488972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.489189 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.489267 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.489393 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.489578 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":497,"threads_started":1}
I20260927 05:17:44.489990 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.475140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.97.3:45439)
0927 05:17:44.476437 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.476447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.476484 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:17:44.476583 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.476591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.476837 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.477029 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:44.479287 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.481310 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.487266 (+ 5956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.488771 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.488913 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.489209 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.489471 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.489636 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":979,"threads_started":1}
I20260927 05:17:44.494251 20632 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 (DEFAULT_TABLE table=test-table [id=90cc704f2947486a9383c5cd3cb45a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:44.494964 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.476812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36759 (local address 127.0.0.1:41488)
0927 05:17:44.487825 (+ 11013us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.487842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.487958 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.488507 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.488514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.488544 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:44.488956 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.488965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.490675 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.490678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.492703 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.492715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.494162 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.494168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.494354 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.494390 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.494487 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.494608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10907,"mutex_wait_us":45,"thread_start_us":10770,"threads_started":1}
I20260927 05:17:44.495107 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.477104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.97.2:36759)
0927 05:17:44.487045 (+ 9941us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.487058 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.487953 (+ 895us) server_negotiation.cc:407] Connection header received
0927 05:17:44.488093 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.488100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.488311 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.488496 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:44.489151 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.490530 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.492889 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.493871 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.493998 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.494263 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.494539 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.494710 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9714,"thread_start_us":9558,"threads_started":1}
I20260927 05:17:44.496176 20632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2f7481e6a24bc1a338cbece33f46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:44.496366 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.476131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56206)
0927 05:17:44.483150 (+ 7019us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.483177 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.483294 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.488138 (+ 4844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.488143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.488165 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:44.488615 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.488631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.491212 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.491219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.493648 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.493673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.495648 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.495652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.495779 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.495799 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.495902 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.496021 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":4,"thread_start_us":308,"threads_started":1}
I20260927 05:17:44.496771 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.476458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.97.1:36691)
0927 05:17:44.487478 (+ 11020us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.487485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.487533 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:17:44.487648 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.487657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.487922 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.488163 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:44.488811 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.490872 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.493859 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.495249 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.495387 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.495551 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.495950 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.496097 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10885,"thread_start_us":10777,"threads_started":1}
I20260927 05:17:44.498813 20502 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 (DEFAULT_TABLE table=test-table [id=90cc704f2947486a9383c5cd3cb45a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:44.500149 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2f7481e6a24bc1a338cbece33f46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:44.501715 20371 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 (DEFAULT_TABLE table=test-table [id=90cc704f2947486a9383c5cd3cb45a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:17:44.502709 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2f7481e6a24bc1a338cbece33f46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:44.511735 20715 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Bootstrap starting.
I20260927 05:17:44.516090 20717 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Bootstrap starting.
I20260927 05:17:44.517315 20715 tablet_bootstrap.cc:654] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:44.517508 20716 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Bootstrap starting.
I20260927 05:17:44.518770 20715 log.cc:824] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:44.519524 20717 tablet_bootstrap.cc:654] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:44.520663 20717 log.cc:824] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:44.520944 20716 tablet_bootstrap.cc:654] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:44.521596 20715 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: No bootstrap required, opened a new log
I20260927 05:17:44.521863 20715 ts_tablet_manager.cc:1397] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 05:17:44.522028 20716 log.cc:824] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:44.523619 20717 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: No bootstrap required, opened a new log
I20260927 05:17:44.523896 20717 ts_tablet_manager.cc:1397] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260927 05:17:44.525054 20716 tablet_bootstrap.cc:492] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: No bootstrap required, opened a new log
I20260927 05:17:44.525307 20716 ts_tablet_manager.cc:1397] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 05:17:44.529814 20715 raft_consensus.cc:348] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.530169 20715 raft_consensus.cc:374] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:44.530269 20715 raft_consensus.cc:729] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc74eb3e64b432d8e794fb2469eceb4, State: Initialized, Role: FOLLOWER
I20260927 05:17:44.530871 20715 consensus_queue.cc:260] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.534267 20717 raft_consensus.cc:348] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.534711 20717 raft_consensus.cc:374] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:44.534847 20717 raft_consensus.cc:729] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b419574cc8664508ae5c0ea8e0110429, State: Initialized, Role: FOLLOWER
I20260927 05:17:44.535348 20697 heartbeater.cc:503] Master 127.0.97.62:42595 was elected leader, sending a full tablet report...
I20260927 05:17:44.535945 20715 ts_tablet_manager.cc:1428] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260927 05:17:44.535676 20716 raft_consensus.cc:348] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.536073 20716 raft_consensus.cc:374] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:44.536229 20716 raft_consensus.cc:729] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe2e55e3f85446595333df4879a1022, State: Initialized, Role: FOLLOWER
I20260927 05:17:44.536629 20717 consensus_queue.cc:260] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.536859 20716 consensus_queue.cc:260] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.544962 20716 ts_tablet_manager.cc:1428] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Time spent starting tablet: real 0.020s user 0.009s sys 0.011s
I20260927 05:17:44.545320 20436 heartbeater.cc:503] Master 127.0.97.62:42595 was elected leader, sending a full tablet report...
I20260927 05:17:44.547116 20567 heartbeater.cc:503] Master 127.0.97.62:42595 was elected leader, sending a full tablet report...
I20260927 05:17:44.547554 20717 ts_tablet_manager.cc:1428] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Time spent starting tablet: real 0.024s user 0.014s sys 0.006s
I20260927 05:17:44.639708 20722 raft_consensus.cc:482] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:17:44.639989 20722 raft_consensus.cc:504] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.641597 20722 leader_election.cc:288] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc74eb3e64b432d8e794fb2469eceb4 (127.0.97.3:45439), b419574cc8664508ae5c0ea8e0110429 (127.0.97.2:36759)
I20260927 05:17:44.648532 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45941 (local address 127.0.97.3:45439)
0927 05:17:44.642182 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.642191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.642738 (+ 547us) server_negotiation.cc:407] Connection header received
0927 05:17:44.642942 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.642948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.643072 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.643166 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:44.643811 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.644767 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.646785 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.647602 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.647749 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.647860 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.648150 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.648268 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:17:44.648572 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.641976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45439 (local address 127.0.97.1:45941)
0927 05:17:44.642613 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.642628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.642803 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.643292 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.643300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.643337 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:44.643653 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.643662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.644902 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.644907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.646571 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.646582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.647781 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.647786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.647944 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.648001 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.648124 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.648255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":354,"threads_started":1}
I20260927 05:17:44.648869 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.642886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59691 (local address 127.0.97.2:36759)
0927 05:17:44.643160 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.643166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.643570 (+ 404us) server_negotiation.cc:407] Connection header received
0927 05:17:44.643776 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.643780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.643840 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.643934 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:17:44.644487 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.645209 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.647133 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.647875 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.648044 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.648139 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.648511 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.648631 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:17:44.648904 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.642802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36759 (local address 127.0.97.1:59691)
0927 05:17:44.643471 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.643485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:17:44.643625 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:17:44.643947 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:17:44.643951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:17:44.643972 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:17:44.644310 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.644318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.645334 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.645342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.646936 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:17:44.646948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.648083 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.648089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:17:44.648310 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.648347 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:17:44.648485 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 05:17:44.648609 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260927 05:17:44.649924 20652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2f7481e6a24bc1a338cbece33f46a1" candidate_uuid: "8fe2e55e3f85446595333df4879a1022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc74eb3e64b432d8e794fb2469eceb4" is_pre_election: true
I20260927 05:17:44.649981 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2f7481e6a24bc1a338cbece33f46a1" candidate_uuid: "8fe2e55e3f85446595333df4879a1022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b419574cc8664508ae5c0ea8e0110429" is_pre_election: true
I20260927 05:17:44.650518 20652 raft_consensus.cc:2454] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe2e55e3f85446595333df4879a1022 in term 0.
I20260927 05:17:44.650518 20522 raft_consensus.cc:2454] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe2e55e3f85446595333df4879a1022 in term 0.
I20260927 05:17:44.651217 20326 leader_election.cc:302] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe2e55e3f85446595333df4879a1022, acc74eb3e64b432d8e794fb2469eceb4; no voters:
I20260927 05:17:44.651531 20722 raft_consensus.cc:2781] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:17:44.651778 20722 raft_consensus.cc:482] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:17:44.651913 20722 raft_consensus.cc:3037] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:44.654052 20722 raft_consensus.cc:504] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.654688 20722 leader_election.cc:288] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [CANDIDATE]: Term 1 election: Requested vote from peers acc74eb3e64b432d8e794fb2469eceb4 (127.0.97.3:45439), b419574cc8664508ae5c0ea8e0110429 (127.0.97.2:36759)
I20260927 05:17:44.655154 20652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2f7481e6a24bc1a338cbece33f46a1" candidate_uuid: "8fe2e55e3f85446595333df4879a1022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc74eb3e64b432d8e794fb2469eceb4"
I20260927 05:17:44.655203 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2f7481e6a24bc1a338cbece33f46a1" candidate_uuid: "8fe2e55e3f85446595333df4879a1022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b419574cc8664508ae5c0ea8e0110429"
I20260927 05:17:44.655345 20652 raft_consensus.cc:3037] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:44.655413 20522 raft_consensus.cc:3037] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:44.657531 20652 raft_consensus.cc:2454] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fe2e55e3f85446595333df4879a1022 in term 1.
I20260927 05:17:44.657667 20522 raft_consensus.cc:2454] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fe2e55e3f85446595333df4879a1022 in term 1.
I20260927 05:17:44.657907 20326 leader_election.cc:302] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe2e55e3f85446595333df4879a1022, acc74eb3e64b432d8e794fb2469eceb4; no voters:
I20260927 05:17:44.658170 20722 raft_consensus.cc:2781] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:44.658672 20722 raft_consensus.cc:686] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 1 LEADER]: Becoming Leader. State: Replica: 8fe2e55e3f85446595333df4879a1022, State: Running, Role: LEADER
I20260927 05:17:44.659219 20722 consensus_queue.cc:237] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } }
I20260927 05:17:44.663944 20250 catalog_manager.cc:5495] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fe2e55e3f85446595333df4879a1022 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8fe2e55e3f85446595333df4879a1022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fe2e55e3f85446595333df4879a1022" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 } health_report { overall_health: UNKNOWN } } }
W20260927 05:17:44.672740 20698 tablet.cc:2369] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:44.719395 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.714549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.97.1:36691)
0927 05:17:44.714742 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.714746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.714763 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:17:44.714901 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.714905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.714988 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.715161 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:17:44.715482 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.716164 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.717449 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.717848 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.717947 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.718338 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:17:44.718806 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:17:44.718914 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.718980 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.719120 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:17:44.727047 20652 raft_consensus.cc:1261] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 1 FOLLOWER]: Refusing update from remote peer 8fe2e55e3f85446595333df4879a1022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:44.727178 20522 raft_consensus.cc:1261] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 1 FOLLOWER]: Refusing update from remote peer 8fe2e55e3f85446595333df4879a1022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:17:44.728048 20726 consensus_queue.cc:1035] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc74eb3e64b432d8e794fb2469eceb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:17:44.728394 20722 consensus_queue.cc:1035] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b419574cc8664508ae5c0ea8e0110429" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 05:17:44.735502 20568 tablet.cc:2369] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:17:44.754172 20250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:44.754377 20250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:17:44.757668 20250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53dd509e9a1747a99076963ffa2cf488: Sending DeleteTablet for 3 replicas of tablet 6e2f7481e6a24bc1a338cbece33f46a1
I20260927 05:17:44.758813 20632 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:44 UTC) from {username='slave'} at 127.0.0.1:47650
I20260927 05:17:44.758888 20502 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:44 UTC) from {username='slave'} at 127.0.0.1:41488
I20260927 05:17:44.758913 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2f7481e6a24bc1a338cbece33f46a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:17:44 UTC) from {username='slave'} at 127.0.0.1:56206
I20260927 05:17:44.761097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20309
I20260927 05:17:44.770816 20741 tablet_replica.cc:330] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: stopping tablet replica
I20260927 05:17:44.771055 20739 tablet_replica.cc:330] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: stopping tablet replica
I20260927 05:17:44.771329 20741 raft_consensus.cc:2229] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:17:44.771428 20739 raft_consensus.cc:2229] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:44.771886 20739 raft_consensus.cc:2258] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:44.774070 20739 ts_tablet_manager.cc:1905] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:44.774369 20741 raft_consensus.cc:2258] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:44.776221 20741 ts_tablet_manager.cc:1905] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:44.780447 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.763894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.0.97.1:36691)
0927 05:17:44.764440 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.764450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.767543 (+ 3093us) server_negotiation.cc:407] Connection header received
0927 05:17:44.767765 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.767773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.767896 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.768021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:44.771100 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.772743 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.775515 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.776393 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.776510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.778274 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:44.778288 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:44.778309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:44.779279 (+ 970us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:44.779351 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:44.779382 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:44.779492 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.779786 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.779916 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260927 05:17:44.780731 20739 ts_tablet_manager.cc:1918] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:44.780879 20739 log.cc:1196] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-2/wal/wals/6e2f7481e6a24bc1a338cbece33f46a1
I20260927 05:17:44.781363 20739 ts_tablet_manager.cc:1939] T 6e2f7481e6a24bc1a338cbece33f46a1 P acc74eb3e64b432d8e794fb2469eceb4: Deleting consensus metadata
I20260927 05:17:44.782341 20431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:44.783126 20236 catalog_manager.cc:4841] TS acc74eb3e64b432d8e794fb2469eceb4 (127.0.97.3:45439): tablet 6e2f7481e6a24bc1a338cbece33f46a1 (table test-table [id=90cc704f2947486a9383c5cd3cb45a3b]) successfully deleted
I20260927 05:17:44.783279 20740 tablet_replica.cc:330] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: stopping tablet replica
I20260927 05:17:44.783695 20740 raft_consensus.cc:2229] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:17:44.784206 20740 raft_consensus.cc:2258] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:17:44.786300 20740 ts_tablet_manager.cc:1905] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:17:44.791502 20740 ts_tablet_manager.cc:1918] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:44.791667 20740 log.cc:1196] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-1/wal/wals/6e2f7481e6a24bc1a338cbece33f46a1
I20260927 05:17:44.792120 20740 ts_tablet_manager.cc:1939] T 6e2f7481e6a24bc1a338cbece33f46a1 P b419574cc8664508ae5c0ea8e0110429: Deleting consensus metadata
I20260927 05:17:44.793521 20236 catalog_manager.cc:4841] TS b419574cc8664508ae5c0ea8e0110429 (127.0.97.2:36759): tablet 6e2f7481e6a24bc1a338cbece33f46a1 (table test-table [id=90cc704f2947486a9383c5cd3cb45a3b]) successfully deleted
I20260927 05:17:44.877575 20741 ts_tablet_manager.cc:1918] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:17:44.878074 20741 log.cc:1196] T 6e2f7481e6a24bc1a338cbece33f46a1 P 8fe2e55e3f85446595333df4879a1022: Deleting WAL directory at /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485951045978-388-0/minicluster-data/ts-0/wal/wals/6e2f7481e6a24bc1a338cbece33f46a1
I20260927 05:17:44.878196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20309
W20260927 05:17:44.896030 20236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36691 (error 108)
I20260927 05:17:44.896732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20440
I20260927 05:17:44.898027 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.897440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56250)
0927 05:17:44.897704 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.897842 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260927 05:17:44.898319 20236 catalog_manager.cc:4584] TS 8fe2e55e3f85446595333df4879a1022 (127.0.97.1:36691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e2f7481e6a24bc1a338cbece33f46a1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
I20260927 05:17:44.903482 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.897577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.97.2:36759)
0927 05:17:44.897730 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:44.897735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:44.897776 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:17:44.897977 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:44.897981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:44.898061 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:44.898191 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:44.898634 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.899445 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.900674 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:44.901071 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:44.901164 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:44.902211 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:44.902221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:44.902236 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:44.902832 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:44.902878 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:44.902898 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:44.902989 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:44.903158 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 05:17:44.903244 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 05:17:44.904156 20562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:44.951939 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.951417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56264)
0927 05:17:44.951637 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.951740 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:17:44.952994 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:44.952609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56278)
0927 05:17:44.952769 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:44.952852 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 05:17:44.993508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20440
I20260927 05:17:45.010994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20570
I20260927 05:17:45.013840 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:45.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56280)
0927 05:17:45.013618 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:45.013692 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:17:45.015978 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:45.015625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56286)
0927 05:17:45.015779 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:45.015857 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:17:45.023957 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:45.011824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.0.97.3:45439)
0927 05:17:45.012024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:45.012030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:45.012051 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:17:45.012225 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:45.012230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:45.012342 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:45.012453 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:45.012832 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:45.013942 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:45.015833 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:45.016449 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:45.016568 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:45.022438 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:45.022454 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:45.022474 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:45.023230 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:45.023300 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:45.023323 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:45.023420 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:45.023551 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 05:17:45.023659 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:17:45.024588 20692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:45.096792 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:45.096227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56290)
0927 05:17:45.096468 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:45.096594 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 05:17:45.097921 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:45.097478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36691 (local address 127.0.0.1:56292)
0927 05:17:45.097691 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:17:45.097787 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 05:17:45.111765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20570
I20260927 05:17:45.128036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20219
I20260927 05:17:45.129114 20280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:45.235850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20219
2026-09-27T05:17:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2722 ms)
[----------] 6 tests from EncryptionPolicyTest (21287 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:45 dist-test-slave-1t6p krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:45 dist-test-slave-1t6p krb5kdc[20750](info): set up 2 sockets
Sep 27 05:17:45 dist-test-slave-1t6p krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20260927 05:17:46.335496 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:46 dist-test-slave-1t6p krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:46.568579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:46.888221 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:46.888490 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:46.888650 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:46.895749 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:46.895848 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:46.895946 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:46.896029 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:46.896088 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:46.905603 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:46.906786 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:46.908066 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:46.919589 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:46.919580 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:46.919632 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:46.920711 20763 server_base.cc:1034] running on GCE node
I20260927 05:17:46.921428 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:46.922720 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:46.923918 20763 hybrid_clock.cc:648] HybridClock initialized: now 1790486266923856 us; error 61 us; skew 500 ppm
I20260927 05:17:46.924247 20763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:46 dist-test-slave-1t6p krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:46.927309 20763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:46.928727 20763 webserver.cc:460] Webserver started at http://127.0.0.1:40521/ using document root <none> and password file <none>
I20260927 05:17:46.929502 20763 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:46.929646 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:46.929973 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:46.932847 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "8757705e37324a3b80dd863427d55ba7"
format_stamp: "Formatted at 2026-09-27 05:17:46 on dist-test-slave-1t6p"
server_key: "c6496188c3af7754e5c41b0895ed9c7d"
server_key_iv: "527fb057ae8cb2c42ff97229788c8c46"
server_key_version: "encryptionkey@0"
I20260927 05:17:46.933578 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "8757705e37324a3b80dd863427d55ba7"
format_stamp: "Formatted at 2026-09-27 05:17:46 on dist-test-slave-1t6p"
server_key: "c6496188c3af7754e5c41b0895ed9c7d"
server_key_iv: "527fb057ae8cb2c42ff97229788c8c46"
server_key_version: "encryptionkey@0"
I20260927 05:17:46.938387 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 05:17:46.941903 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:46.943215 20763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:46.943398 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "8757705e37324a3b80dd863427d55ba7"
format_stamp: "Formatted at 2026-09-27 05:17:46 on dist-test-slave-1t6p"
server_key: "c6496188c3af7754e5c41b0895ed9c7d"
server_key_iv: "527fb057ae8cb2c42ff97229788c8c46"
server_key_version: "encryptionkey@0"
I20260927 05:17:46.943590 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:46.980536 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:46.984783 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:46.985080 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:47.003793 20763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37143
I20260927 05:17:47.003829 20829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37143 every 8 connection(s)
I20260927 05:17:47.005100 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:47.008757 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:47.012300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20763
I20260927 05:17:47.012533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:47.013319 388 external_mini_cluster.cc:1428] Setting key ec634ba2e9855d7ecfee3122bfc7b657
I20260927 05:17:47.021940 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7: Bootstrap starting.
I20260927 05:17:47.025879 20830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:47.027235 20830 log.cc:824] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:17:47 dist-test-slave-1t6p krb5kdc[20750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:17:47.030114 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7: No bootstrap required, opened a new log
I20260927 05:17:47.036934 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:47.015405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55805 (local address 127.0.0.1:37143)
0927 05:17:47.016181 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:47.016192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:47.016239 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:17:47.017849 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:47.017869 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:47.018283 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:47.018595 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:47.020318 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:47.021628 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:47.023525 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:47.024347 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:47.024476 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:47.029524 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:47.029532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:47.029546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:47.029584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:47.032797 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:47.033588 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:47.033594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:47.033600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:47.033765 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:47.034240 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:47.034244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:47.034249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:47.034747 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:47.035126 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:47.035637 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 05:17:47.035933 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":376,"threads_started":1}
I20260927 05:17:47.038239 20830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260927 05:17:47.038632 20830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:47.038744 20830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8757705e37324a3b80dd863427d55ba7, State: Initialized, Role: FOLLOWER
I20260927 05:17:47.039460 20830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260927 05:17:47.039724 20830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:47.039876 20830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:47.040064 20830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:47.040932 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:47.042282 20830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260927 05:17:47.042690 20830 leader_election.cc:302] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8757705e37324a3b80dd863427d55ba7; no voters:
I20260927 05:17:47.043253 20830 leader_election.cc:288] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:47.043394 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:47.044651 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [term 1 LEADER]: Becoming Leader. State: Replica: 8757705e37324a3b80dd863427d55ba7, State: Running, Role: LEADER
I20260927 05:17:47.045369 20830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:47.045288 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260927 05:17:47.049576 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8757705e37324a3b80dd863427d55ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260927 05:17:47.050040 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:47.049839 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8757705e37324a3b80dd863427d55ba7. Latest consensus state: current_term: 1 leader_uuid: "8757705e37324a3b80dd863427d55ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8757705e37324a3b80dd863427d55ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260927 05:17:47.050642 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:47.054948 20847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:47.059643 20847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:47.072335 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:47.047207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34873 (local address 127.0.0.1:37143)
0927 05:17:47.047389 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:47.047400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:47.049580 (+ 2180us) server_negotiation.cc:407] Connection header received
0927 05:17:47.049856 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:47.049864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:47.049966 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:47.050222 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:47.052886 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:47.054421 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:47.057538 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:47.058556 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:47.058716 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:47.062845 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:47.062855 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:47.062863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:47.062916 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:47.067568 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:47.068770 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:47.068779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:47.068784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:47.068973 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:47.069610 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:47.069617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:47.069621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:47.070040 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:47.070298 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:47.071575 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0927 05:17:47.071801 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:17:47.074783 20847 catalog_manager.cc:1347] Generated new cluster ID: 00fc28d59e4c4ebfa6061eaec5d93d70
I20260927 05:17:47.074895 20847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:47.114887 20847 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:47.116127 20847 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:47.132927 20847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8757705e37324a3b80dd863427d55ba7: Generated new TSK 0
I20260927 05:17:47.133671 20847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:47.142189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20763
I20260927 05:17:47.143301 20825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:47.234869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20763
W20260927 05:17:47.252197 20748 connection.cc:535] client connection to 127.0.0.1:37143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37143): Connection reset by peer (error 104)
2026-09-27T05:17:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:47 dist-test-slave-1t6p krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:47 dist-test-slave-1t6p krb5kdc[20862](info): set up 2 sockets
Sep 27 05:17:47 dist-test-slave-1t6p krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260927 05:17:48.323256 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:48 dist-test-slave-1t6p krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:48.545591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:48.865396 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:48.865679 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:48.865839 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:48.872897 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:48.872987 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:48.873096 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:48.873219 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:48.873294 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:48.882740 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:48.884009 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:48.885311 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:48.897382 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:48.897450 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:48.897603 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:48.898427 20875 server_base.cc:1034] running on GCE node
I20260927 05:17:48.899278 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:48.900616 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:48.901854 20875 hybrid_clock.cc:648] HybridClock initialized: now 1790486268901817 us; error 62 us; skew 500 ppm
I20260927 05:17:48.902189 20875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:48 dist-test-slave-1t6p krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:48.905259 20875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:48.906657 20875 webserver.cc:460] Webserver started at http://127.0.0.1:36293/ using document root <none> and password file <none>
I20260927 05:17:48.907401 20875 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:48.907541 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:48.907842 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:48.910630 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "6f582c9c0fa943859ae3dafb1f71579a"
format_stamp: "Formatted at 2026-09-27 05:17:48 on dist-test-slave-1t6p"
server_key: "71209c9da3c20ecd4712701443cecbc6"
server_key_iv: "02bc67197a187f7aad9613644426010a"
server_key_version: "encryptionkey@0"
I20260927 05:17:48.911401 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "6f582c9c0fa943859ae3dafb1f71579a"
format_stamp: "Formatted at 2026-09-27 05:17:48 on dist-test-slave-1t6p"
server_key: "71209c9da3c20ecd4712701443cecbc6"
server_key_iv: "02bc67197a187f7aad9613644426010a"
server_key_version: "encryptionkey@0"
I20260927 05:17:48.916172 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 05:17:48.919359 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:48.920562 20875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:17:48.920739 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "6f582c9c0fa943859ae3dafb1f71579a"
format_stamp: "Formatted at 2026-09-27 05:17:48 on dist-test-slave-1t6p"
server_key: "71209c9da3c20ecd4712701443cecbc6"
server_key_iv: "02bc67197a187f7aad9613644426010a"
server_key_version: "encryptionkey@0"
I20260927 05:17:48.920920 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:48.944995 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:48.949144 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:48.949463 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:48.967350 20875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45225
I20260927 05:17:48.967374 20941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45225 every 8 connection(s)
I20260927 05:17:48.968608 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:48.972193 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:48.976200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20875
I20260927 05:17:48.976388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:48.976926 388 external_mini_cluster.cc:1428] Setting key 5b0ab6b789e824e76d385a3e69e4e1ec
I20260927 05:17:48.984858 20942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a: Bootstrap starting.
I20260927 05:17:48.988394 20942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:48.989507 20942 log.cc:824] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:48.990069 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:48.978657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51725 (local address 127.0.0.1:45225)
0927 05:17:48.979460 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:48.979470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:48.979507 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:17:48.981026 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:48.981046 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:48.981418 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:48.981732 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:48.982339 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:48.983819 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:48.985703 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:48.986353 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:48.986490 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:48.987785 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:48.987792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:48.987805 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:48.988302 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:48.988372 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:48.988398 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:48.988583 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:48.988886 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:17:48.989035 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":405,"threads_started":1}
I20260927 05:17:48.992298 20942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a: No bootstrap required, opened a new log
I20260927 05:17:48.993956 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:49.001478 20942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260927 05:17:49.001864 20942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:49.001968 20942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f582c9c0fa943859ae3dafb1f71579a, State: Initialized, Role: FOLLOWER
I20260927 05:17:49.002481 20942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260927 05:17:49.002733 20942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:49.002889 20942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:49.003103 20942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:49.005252 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:48.998895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39953 (local address 127.0.0.1:45225)
0927 05:17:48.999077 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:48.999086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:48.999442 (+ 356us) server_negotiation.cc:407] Connection header received
0927 05:17:48.999641 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:48.999646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:48.999715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:48.999826 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:49.000334 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:49.001075 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:49.002652 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:49.003296 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:49.003443 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:49.004549 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:49.004555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:49.004565 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:49.004582 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:49.004615 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:49.004629 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:49.004753 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:49.004938 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 05:17:49.004986 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:17:49.005336 20942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260927 05:17:49.005893 20942 leader_election.cc:302] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f582c9c0fa943859ae3dafb1f71579a; no voters:
I20260927 05:17:49.006522 20942 leader_election.cc:288] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:49.006635 20952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:49.007321 20952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [term 1 LEADER]: Becoming Leader. State: Replica: 6f582c9c0fa943859ae3dafb1f71579a, State: Running, Role: LEADER
I20260927 05:17:49.007833 20952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260927 05:17:49.008692 20942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:49.010931 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f582c9c0fa943859ae3dafb1f71579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260927 05:17:49.010883 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f582c9c0fa943859ae3dafb1f71579a. Latest consensus state: current_term: 1 leader_uuid: "6f582c9c0fa943859ae3dafb1f71579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f582c9c0fa943859ae3dafb1f71579a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260927 05:17:49.011368 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:49.011394 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:49.014599 20957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:49.020321 20957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:49.029592 20957 catalog_manager.cc:1347] Generated new cluster ID: 89cc9aa6eb7245b8bcc24050e7338b93
I20260927 05:17:49.029721 20957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:49.041910 20957 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:49.043143 20957 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:49.051636 20957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f582c9c0fa943859ae3dafb1f71579a: Generated new TSK 0
I20260927 05:17:49.052628 20957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:49.067329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20875
W20260927 05:17:49.068717 20937 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51725
W20260927 05:17:49.069432 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:17:49.069597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20875
W20260927 05:17:49.086337 20860 connection.cc:535] client connection to 127.0.0.1:45225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45225): Connection reset by peer (error 104)
2026-09-27T05:17:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:49 dist-test-slave-1t6p krb5kdc[20973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:49 dist-test-slave-1t6p krb5kdc[20973](info): set up 2 sockets
Sep 27 05:17:49 dist-test-slave-1t6p krb5kdc[20973](info): commencing operation
krb5kdc: starting...
W20260927 05:17:50.163470 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:50 dist-test-slave-1t6p krb5kdc[20973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:50.390519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:50.701465 20986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:50.701777 20986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:50.701948 20986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:50.708858 20986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:50.709002 20986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:50.709106 20986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:50.709215 20986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:50.709303 20986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:50.718724 20986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:50.720059 20986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:50.721350 20986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:50.733129 20991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:50.733212 20994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:50.733183 20992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:50.733837 20986 server_base.cc:1034] running on GCE node
I20260927 05:17:50.734541 20986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:50.736039 20986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:50.737246 20986 hybrid_clock.cc:648] HybridClock initialized: now 1790486270737211 us; error 31 us; skew 500 ppm
I20260927 05:17:50.737598 20986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:50 dist-test-slave-1t6p krb5kdc[20973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:50.743818 20986 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:50.745301 20986 webserver.cc:460] Webserver started at http://127.0.0.1:41995/ using document root <none> and password file <none>
I20260927 05:17:50.745985 20986 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:50.746132 20986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:50.746439 20986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:50.749419 20986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "19cbdab2e99546d2809b4e29f87c8058"
format_stamp: "Formatted at 2026-09-27 05:17:50 on dist-test-slave-1t6p"
server_key: "66a9784fc6eee14f141db0a3f2fa3cd4"
server_key_iv: "febdb5489cd37a1aa9e127446135ed58"
server_key_version: "encryptionkey@0"
I20260927 05:17:50.750162 20986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "19cbdab2e99546d2809b4e29f87c8058"
format_stamp: "Formatted at 2026-09-27 05:17:50 on dist-test-slave-1t6p"
server_key: "66a9784fc6eee14f141db0a3f2fa3cd4"
server_key_iv: "febdb5489cd37a1aa9e127446135ed58"
server_key_version: "encryptionkey@0"
I20260927 05:17:50.755134 20986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 05:17:50.758306 21001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:50.759624 20986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:17:50.759860 20986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "19cbdab2e99546d2809b4e29f87c8058"
format_stamp: "Formatted at 2026-09-27 05:17:50 on dist-test-slave-1t6p"
server_key: "66a9784fc6eee14f141db0a3f2fa3cd4"
server_key_iv: "febdb5489cd37a1aa9e127446135ed58"
server_key_version: "encryptionkey@0"
I20260927 05:17:50.760079 20986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:50.765688 20986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:50.769594 20986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:50.769832 20986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:50.788656 20986 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37719
I20260927 05:17:50.788690 21052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37719 every 8 connection(s)
I20260927 05:17:50.790068 20986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:50.793535 21053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:50.799855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 20986
I20260927 05:17:50.800028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:50.800469 388 external_mini_cluster.cc:1428] Setting key 4c835265ecc4cb653e379a89d8d016fe
I20260927 05:17:50.804580 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058: Bootstrap starting.
I20260927 05:17:50.807878 21053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:50.808555 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:50.802319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44131 (local address 127.0.0.1:37719)
0927 05:17:50.803487 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:50.803498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:50.803537 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:17:50.805286 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:50.805305 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:50.805607 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:50.806028 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:50.806995 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:50.807043 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:50.807059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:50.807082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:50.807134 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:50.807154 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:50.807335 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:50.807463 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 05:17:50.807648 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":628,"threads_started":1}
I20260927 05:17:50.809046 21053 log.cc:824] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:50.811987 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058: No bootstrap required, opened a new log
I20260927 05:17:50.812214 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:50.818836 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:50.816250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53231 (local address 127.0.0.1:37719)
0927 05:17:50.816492 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:50.816501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:50.817004 (+ 503us) server_negotiation.cc:407] Connection header received
0927 05:17:50.817185 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:50.817192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:50.817241 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:50.817348 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:50.818221 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:50.818226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:50.818237 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:50.818251 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:50.818287 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:50.818300 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:50.818434 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:50.818529 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 05:17:50.818582 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:17:50.819862 21053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } }
I20260927 05:17:50.820314 21053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:50.820432 21053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cbdab2e99546d2809b4e29f87c8058, State: Initialized, Role: FOLLOWER
I20260927 05:17:50.820940 21053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } }
I20260927 05:17:50.821178 21053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:50.821311 21053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:50.821489 21053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:50.823688 21053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } }
I20260927 05:17:50.824162 21053 leader_election.cc:302] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19cbdab2e99546d2809b4e29f87c8058; no voters:
I20260927 05:17:50.825019 21053 leader_election.cc:288] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:50.825132 21063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:50.825953 21063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [term 1 LEADER]: Becoming Leader. State: Replica: 19cbdab2e99546d2809b4e29f87c8058, State: Running, Role: LEADER
I20260927 05:17:50.826453 21063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } }
I20260927 05:17:50.827108 21053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:50.829165 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19cbdab2e99546d2809b4e29f87c8058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } } }
I20260927 05:17:50.829591 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:50.829141 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19cbdab2e99546d2809b4e29f87c8058. Latest consensus state: current_term: 1 leader_uuid: "19cbdab2e99546d2809b4e29f87c8058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbdab2e99546d2809b4e29f87c8058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37719 } } }
I20260927 05:17:50.830284 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:50.838244 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:50.842468 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:50.850535 21070 catalog_manager.cc:1347] Generated new cluster ID: f7cfff08e7db42938f5c02af61b90167
I20260927 05:17:50.850638 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:50.868076 21070 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:50.868443 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:50.876768 21070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19cbdab2e99546d2809b4e29f87c8058: Generated new TSK 0
I20260927 05:17:50.877442 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:50.890338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20986
W20260927 05:17:50.891393 21048 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44131
W20260927 05:17:50.892032 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 20986: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:17:50.892184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 20986
2026-09-27T05:17:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1821 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:50 dist-test-slave-1t6p krb5kdc[21084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:50 dist-test-slave-1t6p krb5kdc[21084](info): set up 2 sockets
Sep 27 05:17:50 dist-test-slave-1t6p krb5kdc[21084](info): commencing operation
krb5kdc: starting...
W20260927 05:17:51.987273 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.009s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:52.222600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:52.541626 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:52.541898 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:52.541991 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:52.549037 21097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:52.549139 21097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:52.549181 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:52.549242 21097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:52.549278 21097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:52.558600 21097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:52.559782 21097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:52.560943 21097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:52.572511 21102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:52.572449 21103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:52.572551 21105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:52.573618 21097 server_base.cc:1034] running on GCE node
I20260927 05:17:52.574396 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:52.575759 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:52.576982 21097 hybrid_clock.cc:648] HybridClock initialized: now 1790486272576940 us; error 52 us; skew 500 ppm
I20260927 05:17:52.577322 21097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:52.580531 21097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:52.581983 21097 webserver.cc:460] Webserver started at http://127.0.0.1:41787/ using document root <none> and password file <none>
I20260927 05:17:52.582626 21097 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:52.582783 21097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:52.583118 21097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:52.585902 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "b0190a6143a6407892d2ffacb1d6ba03"
format_stamp: "Formatted at 2026-09-27 05:17:52 on dist-test-slave-1t6p"
server_key: "38562b3e4626ace585f0344761ce6f7f"
server_key_iv: "7ecea1138db285791b2c1b838be1a71b"
server_key_version: "encryptionkey@0"
I20260927 05:17:52.586580 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "b0190a6143a6407892d2ffacb1d6ba03"
format_stamp: "Formatted at 2026-09-27 05:17:52 on dist-test-slave-1t6p"
server_key: "38562b3e4626ace585f0344761ce6f7f"
server_key_iv: "7ecea1138db285791b2c1b838be1a71b"
server_key_version: "encryptionkey@0"
I20260927 05:17:52.591471 21097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 05:17:52.594678 21112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:52.595984 21097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:52.596167 21097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "b0190a6143a6407892d2ffacb1d6ba03"
format_stamp: "Formatted at 2026-09-27 05:17:52 on dist-test-slave-1t6p"
server_key: "38562b3e4626ace585f0344761ce6f7f"
server_key_iv: "7ecea1138db285791b2c1b838be1a71b"
server_key_version: "encryptionkey@0"
I20260927 05:17:52.596354 21097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:52.628715 21097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:52.633050 21097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:52.633349 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:52.651760 21097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42201
I20260927 05:17:52.651804 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42201 every 8 connection(s)
I20260927 05:17:52.653071 21097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:52.654078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21097
I20260927 05:17:52.654243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:52.654681 388 external_mini_cluster.cc:1428] Setting key 127c01146c0c86cfafda1e6d4be44555
I20260927 05:17:52.656984 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:52.670526 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03: Bootstrap starting.
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:17:52.673928 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:52.675284 21164 log.cc:824] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:52.677917 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03: No bootstrap required, opened a new log
I20260927 05:17:52.679972 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:52.656670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37091 (local address 127.0.0.1:42201)
0927 05:17:52.659083 (+ 2413us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:52.659096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:52.659142 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:17:52.661569 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:52.661595 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:52.662093 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:52.662527 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:52.663875 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:52.665818 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:52.667081 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:52.668127 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:52.668300 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:52.672468 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:52.672475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:52.672487 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:52.672523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:52.676161 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:52.676831 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:52.676837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:52.676842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:52.676996 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:52.677383 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:52.677387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:52.677389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:52.677878 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:52.678184 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:52.678618 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 05:17:52.679040 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2156,"thread_start_us":1877,"threads_started":1}
I20260927 05:17:52.683884 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:52.686308 21164 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260927 05:17:52.686656 21164 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:52.686785 21164 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190a6143a6407892d2ffacb1d6ba03, State: Initialized, Role: FOLLOWER
I20260927 05:17:52.687381 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260927 05:17:52.687610 21164 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:52.687755 21164 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:52.687937 21164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:52.689960 21164 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260927 05:17:52.690505 21164 leader_election.cc:302] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0190a6143a6407892d2ffacb1d6ba03; no voters:
I20260927 05:17:52.691183 21164 leader_election.cc:288] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:52.691288 21173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:52.692595 21173 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [term 1 LEADER]: Becoming Leader. State: Replica: b0190a6143a6407892d2ffacb1d6ba03, State: Running, Role: LEADER
I20260927 05:17:52.693302 21173 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260927 05:17:52.694597 21164 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:52.696750 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0190a6143a6407892d2ffacb1d6ba03. Latest consensus state: current_term: 1 leader_uuid: "b0190a6143a6407892d2ffacb1d6ba03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } } }
I20260927 05:17:52.697185 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:52.697700 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0190a6143a6407892d2ffacb1d6ba03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0190a6143a6407892d2ffacb1d6ba03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } } }
I20260927 05:17:52.697989 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:52.698167 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:52.711084 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:52.691430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55565 (local address 127.0.0.1:42201)
0927 05:17:52.691644 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:52.691653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:52.692189 (+ 536us) server_negotiation.cc:407] Connection header received
0927 05:17:52.692380 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:52.692388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:52.692484 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:52.692601 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:52.694343 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:52.695220 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:52.697131 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:52.697949 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:52.698052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:52.701886 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:52.701896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:52.701903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:52.701950 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:52.706655 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:52.707843 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:52.707852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:52.707858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:52.708033 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:52.708638 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:52.708646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:52.708652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:52.709195 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:52.709515 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:52.710254 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 05:17:52.710544 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:17:52.715237 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:52.725255 21183 catalog_manager.cc:1347] Generated new cluster ID: a72265889bd441cabeeef5d4811b22ae
I20260927 05:17:52.725364 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:52.740947 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:52.742107 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:52.751850 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0190a6143a6407892d2ffacb1d6ba03: Generated new TSK 0
I20260927 05:17:52.752564 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:52.760502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21097
I20260927 05:17:52.761606 21159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:52.851286 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21097
2026-09-27T05:17:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21196](info): set up 2 sockets
Sep 27 05:17:52 dist-test-slave-1t6p krb5kdc[21196](info): commencing operation
krb5kdc: starting...
W20260927 05:17:53.947381 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:54 dist-test-slave-1t6p krb5kdc[21196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:54.178256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:54.496975 21209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:54.497309 21209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:54.497469 21209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:54.504626 21209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:54.504786 21209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:54.504896 21209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:54.505005 21209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:54.505082 21209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:54.514655 21209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:54.515887 21209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:54.517179 21209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:54.528795 21217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:54.528887 21214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:54.528863 21215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:54.529668 21209 server_base.cc:1034] running on GCE node
I20260927 05:17:54.530419 21209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:54.531791 21209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:54.533005 21209 hybrid_clock.cc:648] HybridClock initialized: now 1790486274532947 us; error 47 us; skew 500 ppm
I20260927 05:17:54.533354 21209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:54 dist-test-slave-1t6p krb5kdc[21196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:54.536480 21209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:54.537876 21209 webserver.cc:460] Webserver started at http://127.0.0.1:36949/ using document root <none> and password file <none>
I20260927 05:17:54.538520 21209 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:54.538645 21209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:54.538935 21209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:54.541724 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "f46f417bcf56492ab84c73233076ac8b"
format_stamp: "Formatted at 2026-09-27 05:17:54 on dist-test-slave-1t6p"
server_key: "bf442ecaf86e351a724a504004e6637e"
server_key_iv: "1ceb5e2dddac540d6470b7862d095876"
server_key_version: "encryptionkey@0"
I20260927 05:17:54.542459 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "f46f417bcf56492ab84c73233076ac8b"
format_stamp: "Formatted at 2026-09-27 05:17:54 on dist-test-slave-1t6p"
server_key: "bf442ecaf86e351a724a504004e6637e"
server_key_iv: "1ceb5e2dddac540d6470b7862d095876"
server_key_version: "encryptionkey@0"
I20260927 05:17:54.547438 21209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 05:17:54.550973 21224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:54.552349 21209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:17:54.552541 21209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "f46f417bcf56492ab84c73233076ac8b"
format_stamp: "Formatted at 2026-09-27 05:17:54 on dist-test-slave-1t6p"
server_key: "bf442ecaf86e351a724a504004e6637e"
server_key_iv: "1ceb5e2dddac540d6470b7862d095876"
server_key_version: "encryptionkey@0"
I20260927 05:17:54.552765 21209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:54.576107 21209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:54.580261 21209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:54.580526 21209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:54.598894 21209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36885
I20260927 05:17:54.598932 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36885 every 8 connection(s)
I20260927 05:17:54.600238 21209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:54.603793 21276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:54.610399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21209
I20260927 05:17:54.610599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:54.611099 388 external_mini_cluster.cc:1428] Setting key 956e04e0d2441f3058607a6a2ecc4954
I20260927 05:17:54.613901 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b: Bootstrap starting.
I20260927 05:17:54.617201 21276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:54.618230 21276 log.cc:824] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:54.621141 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b: No bootstrap required, opened a new log
I20260927 05:17:54.624107 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:54.612905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49391 (local address 127.0.0.1:36885)
0927 05:17:54.613675 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:54.613693 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:54.613732 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:17:54.615538 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:54.615559 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:54.615910 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:54.616200 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:54.616817 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:54.618124 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:54.619923 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:54.620588 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:54.620718 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:54.621904 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:54.621911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:54.621924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:54.622429 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:54.622493 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:54.622515 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:54.622630 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:54.622873 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 05:17:54.623173 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
I20260927 05:17:54.627594 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:54.629166 21276 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } }
I20260927 05:17:54.629487 21276 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:54.629603 21276 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46f417bcf56492ab84c73233076ac8b, State: Initialized, Role: FOLLOWER
I20260927 05:17:54.630312 21276 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } }
I20260927 05:17:54.630596 21276 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:54.630741 21276 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:54.630928 21276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:54.633654 21276 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } }
I20260927 05:17:54.634177 21276 leader_election.cc:302] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46f417bcf56492ab84c73233076ac8b; no voters:
I20260927 05:17:54.634783 21276 leader_election.cc:288] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:54.634924 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:54.635618 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [term 1 LEADER]: Becoming Leader. State: Replica: f46f417bcf56492ab84c73233076ac8b, State: Running, Role: LEADER
I20260927 05:17:54.636210 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } }
I20260927 05:17:54.637979 21276 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:54.639272 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:54.631959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41725 (local address 127.0.0.1:36885)
0927 05:17:54.632126 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:54.632131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:54.632483 (+ 352us) server_negotiation.cc:407] Connection header received
0927 05:17:54.632585 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:54.632590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:54.632665 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:54.632792 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:54.633199 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:54.634262 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:54.636249 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:54.636937 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:54.637064 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:54.638257 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:54.638261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:54.638265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:54.638280 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:54.638312 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:54.638326 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:54.638432 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:54.638846 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 05:17:54.638929 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:17:54.639205 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46f417bcf56492ab84c73233076ac8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } } }
I20260927 05:17:54.639642 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:54.639730 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46f417bcf56492ab84c73233076ac8b. Latest consensus state: current_term: 1 leader_uuid: "f46f417bcf56492ab84c73233076ac8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46f417bcf56492ab84c73233076ac8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36885 } } }
I20260927 05:17:54.640049 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:54.641490 21291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:54.648393 21291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:54.656842 21291 catalog_manager.cc:1347] Generated new cluster ID: 9f3215264207488dad3629c01c599d07
I20260927 05:17:54.656947 21291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:54.677335 21291 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:54.678519 21291 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:54.684569 21291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46f417bcf56492ab84c73233076ac8b: Generated new TSK 0
I20260927 05:17:54.685272 21291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:54.695341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21209
W20260927 05:17:54.696301 21271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49391
W20260927 05:17:54.696961 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21209: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:17:54.697113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21209
2026-09-27T05:17:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:54 dist-test-slave-1t6p krb5kdc[21307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:54 dist-test-slave-1t6p krb5kdc[21307](info): set up 2 sockets
Sep 27 05:17:54 dist-test-slave-1t6p krb5kdc[21307](info): commencing operation
krb5kdc: starting...
W20260927 05:17:55.831310 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:55 dist-test-slave-1t6p krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:56.084472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:56.397475 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:56.397727 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:56.397822 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:56.404407 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:56.404496 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:56.404541 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:56.404599 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:56.404636 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:56.413560 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:56.414659 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:56.415818 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:56.427186 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:56.427126 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:56.427211 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:56.427933 21320 server_base.cc:1034] running on GCE node
I20260927 05:17:56.428664 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:56.429944 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:56.431143 21320 hybrid_clock.cc:648] HybridClock initialized: now 1790486276431112 us; error 44 us; skew 500 ppm
I20260927 05:17:56.431475 21320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:56 dist-test-slave-1t6p krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:56.437978 21320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:56.439414 21320 webserver.cc:460] Webserver started at http://127.0.0.1:40629/ using document root <none> and password file <none>
I20260927 05:17:56.440061 21320 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:56.440201 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:56.440500 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:56.443333 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "d0ae4971fef440079d2e88ae41fbe47c"
format_stamp: "Formatted at 2026-09-27 05:17:56 on dist-test-slave-1t6p"
server_key: "8815b64e5c3def8541effc9f79f21f2d"
server_key_iv: "5fcb2b9ecfe961e7c642732a99b74766"
server_key_version: "encryptionkey@0"
I20260927 05:17:56.444067 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0ae4971fef440079d2e88ae41fbe47c"
format_stamp: "Formatted at 2026-09-27 05:17:56 on dist-test-slave-1t6p"
server_key: "8815b64e5c3def8541effc9f79f21f2d"
server_key_iv: "5fcb2b9ecfe961e7c642732a99b74766"
server_key_version: "encryptionkey@0"
I20260927 05:17:56.448843 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:17:56.452157 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:56.453500 21320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:17:56.453722 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "d0ae4971fef440079d2e88ae41fbe47c"
format_stamp: "Formatted at 2026-09-27 05:17:56 on dist-test-slave-1t6p"
server_key: "8815b64e5c3def8541effc9f79f21f2d"
server_key_iv: "5fcb2b9ecfe961e7c642732a99b74766"
server_key_version: "encryptionkey@0"
I20260927 05:17:56.453940 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:56.459843 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:56.463963 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:56.464280 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:56.483541 21320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44615
I20260927 05:17:56.483582 21386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44615 every 8 connection(s)
I20260927 05:17:56.484965 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:56.488838 21387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:17:56.495325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21320
I20260927 05:17:56.495522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:56.495994 388 external_mini_cluster.cc:1428] Setting key a23f9c647617c5af6bc5d6b553d83507
I20260927 05:17:56.500121 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c: Bootstrap starting.
I20260927 05:17:56.503945 21387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:56.504036 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:56.497802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38593 (local address 127.0.0.1:44615)
0927 05:17:56.498667 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:56.498679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:56.498720 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:17:56.500609 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:56.500631 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:56.500959 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:56.501282 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:56.502436 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:56.502442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:56.502454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:56.502470 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:56.502527 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:56.502548 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:56.502715 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:56.502857 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 05:17:56.503084 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":418,"threads_started":1}
I20260927 05:17:56.505426 21387 log.cc:824] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:56.508208 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:56.508522 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c: No bootstrap required, opened a new log
I20260927 05:17:56.514916 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:56.512489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57757 (local address 127.0.0.1:44615)
0927 05:17:56.512672 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:56.512678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:56.512996 (+ 318us) server_negotiation.cc:407] Connection header received
0927 05:17:56.513151 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:56.513155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:56.513196 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:56.513280 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:56.514234 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:56.514239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:56.514243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:17:56.514261 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:56.514295 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:17:56.514312 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:56.514407 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:56.514554 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:17:56.514595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:17:56.517688 21387 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260927 05:17:56.518075 21387 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:56.518198 21387 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ae4971fef440079d2e88ae41fbe47c, State: Initialized, Role: FOLLOWER
I20260927 05:17:56.518779 21387 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260927 05:17:56.519076 21387 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:56.519243 21387 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:56.519428 21387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:56.521703 21387 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260927 05:17:56.522234 21387 leader_election.cc:302] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ae4971fef440079d2e88ae41fbe47c; no voters:
I20260927 05:17:56.522784 21387 leader_election.cc:288] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:56.522919 21397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:56.523694 21397 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [term 1 LEADER]: Becoming Leader. State: Replica: d0ae4971fef440079d2e88ae41fbe47c, State: Running, Role: LEADER
I20260927 05:17:56.524230 21397 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260927 05:17:56.525017 21387 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:56.527221 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ae4971fef440079d2e88ae41fbe47c. Latest consensus state: current_term: 1 leader_uuid: "d0ae4971fef440079d2e88ae41fbe47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260927 05:17:56.527722 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:56.527570 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ae4971fef440079d2e88ae41fbe47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ae4971fef440079d2e88ae41fbe47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260927 05:17:56.527920 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:56.533913 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:56.539611 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:56.548739 21404 catalog_manager.cc:1347] Generated new cluster ID: f08bfec8f06f450c994590d21e792fb8
I20260927 05:17:56.548856 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:56.575536 21404 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:56.575974 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:56.583443 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ae4971fef440079d2e88ae41fbe47c: Generated new TSK 0
I20260927 05:17:56.584228 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:56.598510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21320
W20260927 05:17:56.599824 21382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38593
W20260927 05:17:56.600553 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21320: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:17:56.600710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21320
2026-09-27T05:17:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:56 dist-test-slave-1t6p krb5kdc[21418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:56 dist-test-slave-1t6p krb5kdc[21418](info): set up 2 sockets
Sep 27 05:17:56 dist-test-slave-1t6p krb5kdc[21418](info): commencing operation
krb5kdc: starting...
W20260927 05:17:57.695397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:57 dist-test-slave-1t6p krb5kdc[21418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:57.926151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:17:58.242962 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:17:58.243295 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:17:58.243458 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:17:58.250468 21431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:17:58.250599 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:17:58.250702 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:17:58.250838 21431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:17:58.250921 21431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:17:58.260249 21431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:17:58.261387 21431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:17:58.262564 21431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:17:58.274044 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:58.273980 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:17:58.274103 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:17:58.275228 21431 server_base.cc:1034] running on GCE node
I20260927 05:17:58.275943 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:17:58.277189 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:17:58.278411 21431 hybrid_clock.cc:648] HybridClock initialized: now 1790486278278351 us; error 54 us; skew 500 ppm
I20260927 05:17:58.278771 21431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:17:58 dist-test-slave-1t6p krb5kdc[21418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:17:58.281894 21431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:17:58.283250 21431 webserver.cc:460] Webserver started at http://127.0.0.1:33803/ using document root <none> and password file <none>
I20260927 05:17:58.283855 21431 fs_manager.cc:359] Metadata directory not provided
I20260927 05:17:58.283950 21431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:17:58.284183 21431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:17:58.286882 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "7d34f2d5bea34187a797b2416165ab93"
format_stamp: "Formatted at 2026-09-27 05:17:58 on dist-test-slave-1t6p"
server_key: "f1f2195aa66f0dfe4129c04ac40065ef"
server_key_iv: "f1f9ba769ddc2a8d6112935e5363703f"
server_key_version: "encryptionkey@0"
I20260927 05:17:58.287628 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d34f2d5bea34187a797b2416165ab93"
format_stamp: "Formatted at 2026-09-27 05:17:58 on dist-test-slave-1t6p"
server_key: "f1f2195aa66f0dfe4129c04ac40065ef"
server_key_iv: "f1f9ba769ddc2a8d6112935e5363703f"
server_key_version: "encryptionkey@0"
I20260927 05:17:58.292294 21431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 05:17:58.295516 21446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:17:58.296768 21431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:17:58.296947 21431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "7d34f2d5bea34187a797b2416165ab93"
format_stamp: "Formatted at 2026-09-27 05:17:58 on dist-test-slave-1t6p"
server_key: "f1f2195aa66f0dfe4129c04ac40065ef"
server_key_iv: "f1f9ba769ddc2a8d6112935e5363703f"
server_key_version: "encryptionkey@0"
I20260927 05:17:58.297133 21431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:17:58.312176 21431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:17:58.316079 21431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:17:58.316363 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:17:58.334756 21431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33263
I20260927 05:17:58.334787 21497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33263 every 8 connection(s)
I20260927 05:17:58.336146 21431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:17:58.336867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21431
I20260927 05:17:58.337085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:17:58.337611 388 external_mini_cluster.cc:1428] Setting key dbd833708c4527d46b03ea60ee2a4fc5
I20260927 05:17:58.340472 21498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:17:58 dist-test-slave-1t6p krb5kdc[21418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:17:58.353427 21498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93: Bootstrap starting.
I20260927 05:17:58.357389 21498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93: Neither blocks nor log segments found. Creating new log.
I20260927 05:17:58.358628 21498 log.cc:824] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93: Log is configured to *not* fsync() on all Append() calls
I20260927 05:17:58.361594 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:58.339539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:39933 (local address 127.0.0.1:33263)
0927 05:17:58.340301 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:58.340313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:58.340351 (+ 38us) server_negotiation.cc:407] Connection header received
0927 05:17:58.341956 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:58.341976 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:58.342349 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:58.342652 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:58.344389 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:58.345726 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:58.347542 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:58.348180 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:58.348315 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:58.352987 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:58.352997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:58.353016 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:58.353067 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:58.357035 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:58.357859 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:58.357867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:58.357874 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:58.358061 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:58.358521 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:58.358525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:58.358528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:58.359228 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:58.359537 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:58.360024 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 05:17:58.360349 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":355,"threads_started":1}
I20260927 05:17:58.361876 21498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93: No bootstrap required, opened a new log
I20260927 05:17:58.365417 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:17:58.370448 21498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260927 05:17:58.370798 21498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:17:58.370898 21498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d34f2d5bea34187a797b2416165ab93, State: Initialized, Role: FOLLOWER
I20260927 05:17:58.371547 21498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260927 05:17:58.371776 21498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:17:58.371927 21498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:17:58.372090 21498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:17:58.374262 21498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260927 05:17:58.374790 21498 leader_election.cc:302] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d34f2d5bea34187a797b2416165ab93; no voters:
I20260927 05:17:58.375442 21498 leader_election.cc:288] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:17:58.375551 21508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:17:58.377738 21508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [term 1 LEADER]: Becoming Leader. State: Replica: 7d34f2d5bea34187a797b2416165ab93, State: Running, Role: LEADER
I20260927 05:17:58.377962 21498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:17:58.378412 21508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260927 05:17:58.381680 21510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d34f2d5bea34187a797b2416165ab93. Latest consensus state: current_term: 1 leader_uuid: "7d34f2d5bea34187a797b2416165ab93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } } }
I20260927 05:17:58.382129 21510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:58.383082 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d34f2d5bea34187a797b2416165ab93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d34f2d5bea34187a797b2416165ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } } }
I20260927 05:17:58.384112 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93 [sys.catalog]: This master's current role is: LEADER
I20260927 05:17:58.386530 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:17:58.369538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:51075 (local address 127.0.0.1:33263)
0927 05:17:58.369726 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:17:58.369734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:17:58.370114 (+ 380us) server_negotiation.cc:407] Connection header received
0927 05:17:58.370357 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:17:58.370361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:17:58.370421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:17:58.370529 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:17:58.372182 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:58.372997 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:58.374211 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:17:58.374708 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:17:58.374862 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:17:58.377595 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:17:58.377603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:17:58.377613 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:17:58.377660 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:17:58.381587 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:58.382620 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:58.382624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:58.382626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:58.383076 (+ 450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:17:58.383667 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:17:58.383675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:17:58.383689 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:17:58.384187 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:17:58.384519 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 05:17:58.385864 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0927 05:17:58.386088 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:17:58.390709 21516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:17:58.394728 21516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:17:58.402958 21516 catalog_manager.cc:1347] Generated new cluster ID: 86a1f88f4cf2466a806d16e56f94eefd
I20260927 05:17:58.403081 21516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:17:58.415271 21516 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:17:58.416467 21516 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:17:58.421491 21516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d34f2d5bea34187a797b2416165ab93: Generated new TSK 0
I20260927 05:17:58.422339 21516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:17:58.428140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21431
I20260927 05:17:58.430045 21493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:17:58.516050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21431
W20260927 05:17:58.533375 21416 connection.cc:535] client connection to 127.0.0.1:33263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33263): Connection reset by peer (error 104)
2026-09-27T05:17:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:17:58 dist-test-slave-1t6p krb5kdc[21531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:17:58 dist-test-slave-1t6p krb5kdc[21531](info): set up 2 sockets
Sep 27 05:17:58 dist-test-slave-1t6p krb5kdc[21531](info): commencing operation
krb5kdc: starting...
W20260927 05:17:59.627329 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:17:59 dist-test-slave-1t6p krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:17:59.861781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:00.182523 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:00.182844 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:00.183032 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:00.190135 21544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:00.190280 21544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:00.190397 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:00.190498 21544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:00.190580 21544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:00.200291 21544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:00.201557 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:00.202832 21544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:00.213984 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:00.214067 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:00.214035 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:00.214942 21544 server_base.cc:1034] running on GCE node
I20260927 05:18:00.215691 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:00.216949 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:00.218142 21544 hybrid_clock.cc:648] HybridClock initialized: now 1790486280218106 us; error 23 us; skew 500 ppm
I20260927 05:18:00.218477 21544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:00 dist-test-slave-1t6p krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:00.221830 21544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:18:00.223241 21544 webserver.cc:460] Webserver started at http://127.0.0.1:45767/ using document root <none> and password file <none>
I20260927 05:18:00.223918 21544 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:00.224066 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:00.224370 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:00.227236 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "17907eb05b6441b29821bc6cc157e6b3"
format_stamp: "Formatted at 2026-09-27 05:18:00 on dist-test-slave-1t6p"
server_key: "a8bc12eb0ab15c3efbc05f5d5a1b7d52"
server_key_iv: "8c02d4946838d2dc1cacb4dcefd86ba2"
server_key_version: "encryptionkey@0"
I20260927 05:18:00.227962 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "17907eb05b6441b29821bc6cc157e6b3"
format_stamp: "Formatted at 2026-09-27 05:18:00 on dist-test-slave-1t6p"
server_key: "a8bc12eb0ab15c3efbc05f5d5a1b7d52"
server_key_iv: "8c02d4946838d2dc1cacb4dcefd86ba2"
server_key_version: "encryptionkey@0"
I20260927 05:18:00.232720 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:18:00.235859 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:00.237063 21544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:18:00.237237 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "17907eb05b6441b29821bc6cc157e6b3"
format_stamp: "Formatted at 2026-09-27 05:18:00 on dist-test-slave-1t6p"
server_key: "a8bc12eb0ab15c3efbc05f5d5a1b7d52"
server_key_iv: "8c02d4946838d2dc1cacb4dcefd86ba2"
server_key_version: "encryptionkey@0"
I20260927 05:18:00.237452 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:00.258332 21544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:00.261981 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:00.262254 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:00.279644 21544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42689
I20260927 05:18:00.279683 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42689 every 8 connection(s)
I20260927 05:18:00.280848 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:00.282697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21544
I20260927 05:18:00.282896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:00.283463 388 external_mini_cluster.cc:1428] Setting key 829638c1209b7614d1ea757770315778
I20260927 05:18:00.284973 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:00.295696 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:00.285164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:54437 (local address 127.0.0.1:42689)
0927 05:18:00.285868 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:00.285879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:00.285916 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:18:00.287588 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:00.287607 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:00.287972 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:00.288348 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:00.288871 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:00.290139 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:00.291489 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:00.292118 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:00.292305 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:00.293525 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:00.293535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:00.293547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:00.294065 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:00.294142 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:00.294165 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:00.294349 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:00.294587 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 05:18:00.294775 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":292,"threads_started":1}
I20260927 05:18:00.295957 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: Bootstrap starting.
I20260927 05:18:00.299301 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:00.299314 21611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:00.300220 21611 log.cc:824] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:00.302896 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: No bootstrap required, opened a new log
I20260927 05:18:00.310491 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:00.303975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:33381 (local address 127.0.0.1:42689)
0927 05:18:00.304168 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:00.304173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:00.304483 (+ 310us) server_negotiation.cc:407] Connection header received
0927 05:18:00.304641 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:00.304645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:00.304700 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:00.304795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:00.305391 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:00.306174 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:00.308048 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:00.308491 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:00.308580 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:00.309754 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:00.309759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:00.309762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:00.309780 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:00.309812 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:00.309826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:00.309907 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:00.310111 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 05:18:00.310148 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:18:00.312381 21611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } }
I20260927 05:18:00.312744 21611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:00.312853 21611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17907eb05b6441b29821bc6cc157e6b3, State: Initialized, Role: FOLLOWER
I20260927 05:18:00.313369 21611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } }
I20260927 05:18:00.313601 21611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:00.313740 21611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:00.313917 21611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:00.316010 21611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } }
I20260927 05:18:00.316483 21611 leader_election.cc:302] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17907eb05b6441b29821bc6cc157e6b3; no voters:
I20260927 05:18:00.316999 21611 leader_election.cc:288] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:00.317132 21621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:00.318300 21621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 17907eb05b6441b29821bc6cc157e6b3, State: Running, Role: LEADER
I20260927 05:18:00.318818 21621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } }
I20260927 05:18:00.319072 21611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:00.321498 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17907eb05b6441b29821bc6cc157e6b3. Latest consensus state: current_term: 1 leader_uuid: "17907eb05b6441b29821bc6cc157e6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } } }
I20260927 05:18:00.321861 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:00.322645 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17907eb05b6441b29821bc6cc157e6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17907eb05b6441b29821bc6cc157e6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42689 } } }
I20260927 05:18:00.323083 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3 [sys.catalog]: This master's current role is: LEADER
W20260927 05:18:00.333735 21637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:00.333846 21637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:00.334017 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:00.335737 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:00.343222 21627 catalog_manager.cc:1347] Generated new cluster ID: cf37364ace9b46ea95a6810dfeb5a411
I20260927 05:18:00.343405 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:00.350478 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:00.351903 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:00.359237 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17907eb05b6441b29821bc6cc157e6b3: Generated new TSK 0
I20260927 05:18:00.359925 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:00.366710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21544
W20260927 05:18:00.367820 21606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:54437
W20260927 05:18:00.368535 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:00.368676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21544
W20260927 05:18:00.385274 21529 connection.cc:535] client connection to 127.0.0.1:42689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42689): Connection reset by peer (error 104)
2026-09-27T05:18:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:00 dist-test-slave-1t6p krb5kdc[21642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:00 dist-test-slave-1t6p krb5kdc[21642](info): set up 2 sockets
Sep 27 05:18:00 dist-test-slave-1t6p krb5kdc[21642](info): commencing operation
krb5kdc: starting...
W20260927 05:18:01.463447 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:01 dist-test-slave-1t6p krb5kdc[21642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:18:01.691869 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:02.007577 21655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:02.007897 21655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:02.008054 21655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:02.015107 21655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:02.015249 21655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:02.015367 21655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:02.015465 21655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:02.015551 21655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:02.024803 21655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:02.026029 21655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:02.027309 21655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:02.039083 21660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:02.039073 21661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:02.039084 21663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:02.040057 21655 server_base.cc:1034] running on GCE node
I20260927 05:18:02.040827 21655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:02.042145 21655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:02.043346 21655 hybrid_clock.cc:648] HybridClock initialized: now 1790486282043315 us; error 31 us; skew 500 ppm
I20260927 05:18:02.043679 21655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:02 dist-test-slave-1t6p krb5kdc[21642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:02.053434 21655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:18:02.054877 21655 webserver.cc:460] Webserver started at http://127.0.0.1:39327/ using document root <none> and password file <none>
I20260927 05:18:02.055572 21655 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:02.055724 21655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:02.056025 21655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:02.059212 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "1ddb5f0a504f4e1e81b46230df982fff"
format_stamp: "Formatted at 2026-09-27 05:18:02 on dist-test-slave-1t6p"
server_key: "a2c5396427f29a1423496d07e079c5f4"
server_key_iv: "5dc392d95bd761822147a077a7570e15"
server_key_version: "encryptionkey@0"
I20260927 05:18:02.060047 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "1ddb5f0a504f4e1e81b46230df982fff"
format_stamp: "Formatted at 2026-09-27 05:18:02 on dist-test-slave-1t6p"
server_key: "a2c5396427f29a1423496d07e079c5f4"
server_key_iv: "5dc392d95bd761822147a077a7570e15"
server_key_version: "encryptionkey@0"
I20260927 05:18:02.065101 21655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:18:02.068472 21670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:02.069784 21655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:18:02.069979 21655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "1ddb5f0a504f4e1e81b46230df982fff"
format_stamp: "Formatted at 2026-09-27 05:18:02 on dist-test-slave-1t6p"
server_key: "a2c5396427f29a1423496d07e079c5f4"
server_key_iv: "5dc392d95bd761822147a077a7570e15"
server_key_version: "encryptionkey@0"
I20260927 05:18:02.070173 21655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:02.075742 21655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:02.079890 21655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:02.080186 21655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:02.097996 21655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34931
I20260927 05:18:02.098078 21721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34931 every 8 connection(s)
I20260927 05:18:02.099404 21655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:02.101862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21655
I20260927 05:18:02.102046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:02.102551 388 external_mini_cluster.cc:1428] Setting key 88ef134e0dd8b03e0963472dca53efde
I20260927 05:18:02.103415 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:02.110786 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:02.104328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:41331 (local address 127.0.0.1:34931)
0927 05:18:02.105091 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:02.105108 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:02.105156 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:18:02.106967 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:02.106995 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:02.107464 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:02.107901 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:02.108867 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:02.108874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:02.108890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:02.108916 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:02.109001 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:02.109025 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:02.109236 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:02.109369 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:18:02.109568 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":311,"threads_started":1}
I20260927 05:18:02.114688 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:02.117858 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff: Bootstrap starting.
I20260927 05:18:02.121838 21722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:02.122515 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:02.120011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:32991 (local address 127.0.0.1:34931)
0927 05:18:02.120176 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:02.120187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:02.120610 (+ 423us) server_negotiation.cc:407] Connection header received
0927 05:18:02.120752 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:02.120757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:02.120815 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:02.120935 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:02.121848 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:02.121855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:02.121862 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:02.121882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:02.121930 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:02.121951 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:02.122066 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:02.122156 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 05:18:02.122224 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:18:02.123153 21722 log.cc:824] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:02.125805 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff: No bootstrap required, opened a new log
I20260927 05:18:02.133140 21722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 05:18:02.133512 21722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:02.133646 21722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ddb5f0a504f4e1e81b46230df982fff, State: Initialized, Role: FOLLOWER
I20260927 05:18:02.134258 21722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 05:18:02.134503 21722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:02.134652 21722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:02.134822 21722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:02.136943 21722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 05:18:02.137455 21722 leader_election.cc:302] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ddb5f0a504f4e1e81b46230df982fff; no voters:
I20260927 05:18:02.138093 21722 leader_election.cc:288] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:02.138211 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:02.138893 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [term 1 LEADER]: Becoming Leader. State: Replica: 1ddb5f0a504f4e1e81b46230df982fff, State: Running, Role: LEADER
I20260927 05:18:02.139495 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 05:18:02.140341 21722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:02.142309 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ddb5f0a504f4e1e81b46230df982fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260927 05:18:02.142681 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:02.142554 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ddb5f0a504f4e1e81b46230df982fff. Latest consensus state: current_term: 1 leader_uuid: "1ddb5f0a504f4e1e81b46230df982fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb5f0a504f4e1e81b46230df982fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260927 05:18:02.142911 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:02.144498 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:02.150607 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:02.158790 21739 catalog_manager.cc:1347] Generated new cluster ID: 1cba22ed8d864ee1a6f22ad46fb625c0
I20260927 05:18:02.158893 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:02.188613 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:02.189131 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:02.194306 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ddb5f0a504f4e1e81b46230df982fff: Generated new TSK 0
I20260927 05:18:02.195071 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:02.202138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21655
W20260927 05:18:02.203224 21717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:41331
W20260927 05:18:02.203910 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:02.204051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21655
2026-09-27T05:18:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:02 dist-test-slave-1t6p krb5kdc[21753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:02 dist-test-slave-1t6p krb5kdc[21753](info): set up 2 sockets
Sep 27 05:18:02 dist-test-slave-1t6p krb5kdc[21753](info): commencing operation
krb5kdc: starting...
W20260927 05:18:03.291371 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:03 dist-test-slave-1t6p krb5kdc[21753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:18:03.524156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:03.851109 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:03.851411 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:03.851562 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:03.858593 21766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:03.858739 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:03.858840 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:03.858942 21766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:03.859040 21766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:03.868383 21766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:03.869616 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:03.870863 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:03.882052 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:03.882015 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:03.882076 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:03.883248 21766 server_base.cc:1034] running on GCE node
I20260927 05:18:03.884038 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:03.885324 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:03.886619 21766 hybrid_clock.cc:648] HybridClock initialized: now 1790486283886552 us; error 46 us; skew 500 ppm
I20260927 05:18:03.887104 21766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:03 dist-test-slave-1t6p krb5kdc[21753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:03.890199 21766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:18:03.891770 21766 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
I20260927 05:18:03.892439 21766 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:03.892608 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:03.892941 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:03.895972 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "7bf177b925c945cc9566eba622a44cee"
format_stamp: "Formatted at 2026-09-27 05:18:03 on dist-test-slave-1t6p"
server_key: "c5160f43f49b42afe8196b79673184bc"
server_key_iv: "db0ff6355fff38ebf65a4604d9fe5ea7"
server_key_version: "encryptionkey@0"
I20260927 05:18:03.896715 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "7bf177b925c945cc9566eba622a44cee"
format_stamp: "Formatted at 2026-09-27 05:18:03 on dist-test-slave-1t6p"
server_key: "c5160f43f49b42afe8196b79673184bc"
server_key_iv: "db0ff6355fff38ebf65a4604d9fe5ea7"
server_key_version: "encryptionkey@0"
I20260927 05:18:03.901567 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:18:03.904867 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:03.906104 21766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 05:18:03.906286 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "7bf177b925c945cc9566eba622a44cee"
format_stamp: "Formatted at 2026-09-27 05:18:03 on dist-test-slave-1t6p"
server_key: "c5160f43f49b42afe8196b79673184bc"
server_key_iv: "db0ff6355fff38ebf65a4604d9fe5ea7"
server_key_version: "encryptionkey@0"
I20260927 05:18:03.906467 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:03.937377 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:03.941718 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:03.942023 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:03.960170 21766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37009
I20260927 05:18:03.960214 21832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37009 every 8 connection(s)
I20260927 05:18:03.961469 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:03.964869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21766
I20260927 05:18:03.965049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:03.965147 21833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:03.965631 388 external_mini_cluster.cc:1428] Setting key ef3c2569deb16885c23341534d1bae96
I20260927 05:18:03.977020 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee: Bootstrap starting.
Sep 27 05:18:03 dist-test-slave-1t6p krb5kdc[21753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:18:03.980511 21833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:03.981866 21833 log.cc:824] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:03.984995 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee: No bootstrap required, opened a new log
I20260927 05:18:03.988234 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:03.967379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:51789 (local address 127.0.0.1:37009)
0927 05:18:03.968096 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:03.968107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:03.968150 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:18:03.969872 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:03.969892 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:03.970276 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:03.970644 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:03.972031 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:03.973254 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:03.974489 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:03.975245 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:03.975456 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:03.979919 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:03.979926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:03.979938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:03.979975 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:03.983657 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:03.984420 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:03.984430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:03.984440 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:03.984636 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:03.985068 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:03.985073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:03.985076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:03.985763 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:03.986101 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:03.986565 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 05:18:03.986898 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":319,"threads_started":1}
I20260927 05:18:03.992112 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:03.994357 21833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 05:18:03.994699 21833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:03.994823 21833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf177b925c945cc9566eba622a44cee, State: Initialized, Role: FOLLOWER
I20260927 05:18:03.995544 21833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 05:18:03.995824 21833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:03.995967 21833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:03.996137 21833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:03.999500 21833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 05:18:04.000039 21833 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf177b925c945cc9566eba622a44cee; no voters:
I20260927 05:18:04.000671 21833 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:04.000782 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:04.002156 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [term 1 LEADER]: Becoming Leader. State: Replica: 7bf177b925c945cc9566eba622a44cee, State: Running, Role: LEADER
I20260927 05:18:04.002805 21833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:04.002804 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 05:18:04.005498 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf177b925c945cc9566eba622a44cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20260927 05:18:04.005867 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:04.017307 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:03.996498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:48915 (local address 127.0.0.1:37009)
0927 05:18:03.996709 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:03.996719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:03.997088 (+ 369us) server_negotiation.cc:407] Connection header received
0927 05:18:03.997267 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:03.997270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:03.997327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:03.997406 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:03.999259 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:04.000061 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:04.001990 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:04.002709 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:04.002849 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:04.006420 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:04.006431 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:04.006438 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:04.006492 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:04.011519 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:04.013344 (+ 1825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:04.013353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:04.013359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:04.013534 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:04.014281 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:04.014289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:04.014295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:04.014794 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:04.015168 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:04.016561 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0927 05:18:04.016787 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:18:04.021332 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:04.021137 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf177b925c945cc9566eba622a44cee. Latest consensus state: current_term: 1 leader_uuid: "7bf177b925c945cc9566eba622a44cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf177b925c945cc9566eba622a44cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20260927 05:18:04.021603 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:04.027079 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:04.035964 21848 catalog_manager.cc:1347] Generated new cluster ID: 83a494a7aac44ab7b7a2df46c3228a33
I20260927 05:18:04.036063 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:04.049654 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:04.050841 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:04.059131 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf177b925c945cc9566eba622a44cee: Generated new TSK 0
I20260927 05:18:04.059886 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:04.066423 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21766
I20260927 05:18:04.068778 21828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:04.155937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21766
W20260927 05:18:04.169819 21751 connection.cc:535] client connection to 127.0.0.1:37009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37009): Connection reset by peer (error 104)
2026-09-27T05:18:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:04 dist-test-slave-1t6p krb5kdc[21866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:04 dist-test-slave-1t6p krb5kdc[21866](info): set up 2 sockets
Sep 27 05:18:04 dist-test-slave-1t6p krb5kdc[21866](info): commencing operation
krb5kdc: starting...
W20260927 05:18:05.243466 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.010s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:05 dist-test-slave-1t6p krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:18:05.472183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:05.790429 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:05.790748 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:05.790921 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:05.798137 21879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:05.798280 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:05.798377 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:05.798488 21879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:05.798573 21879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:05.808000 21879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:05.809229 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:05.810490 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:05.823329 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:05.823292 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:05.823221 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:05.824199 21879 server_base.cc:1034] running on GCE node
I20260927 05:18:05.825050 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:05.826450 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:05.827703 21879 hybrid_clock.cc:648] HybridClock initialized: now 1790486285827675 us; error 54 us; skew 500 ppm
I20260927 05:18:05.828052 21879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:05 dist-test-slave-1t6p krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:05.831212 21879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:18:05.832923 21879 webserver.cc:460] Webserver started at http://127.0.0.1:42113/ using document root <none> and password file <none>
I20260927 05:18:05.833602 21879 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:05.833745 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:05.834081 21879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:05.837006 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "f241d5e689514a1c83a43c51a41e30eb"
format_stamp: "Formatted at 2026-09-27 05:18:05 on dist-test-slave-1t6p"
server_key: "8fb8cfb83592c80260c485725aa02b47"
server_key_iv: "97cf9ab282618b9af4c15b83ef898c46"
server_key_version: "encryptionkey@0"
I20260927 05:18:05.837729 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "f241d5e689514a1c83a43c51a41e30eb"
format_stamp: "Formatted at 2026-09-27 05:18:05 on dist-test-slave-1t6p"
server_key: "8fb8cfb83592c80260c485725aa02b47"
server_key_iv: "97cf9ab282618b9af4c15b83ef898c46"
server_key_version: "encryptionkey@0"
I20260927 05:18:05.843357 21879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:18:05.846695 21894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:05.847998 21879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:18:05.848172 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "f241d5e689514a1c83a43c51a41e30eb"
format_stamp: "Formatted at 2026-09-27 05:18:05 on dist-test-slave-1t6p"
server_key: "8fb8cfb83592c80260c485725aa02b47"
server_key_iv: "97cf9ab282618b9af4c15b83ef898c46"
server_key_version: "encryptionkey@0"
I20260927 05:18:05.848353 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:05.873680 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:05.877744 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:05.878017 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:05.895938 21879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38699
I20260927 05:18:05.895968 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38699 every 8 connection(s)
I20260927 05:18:05.897429 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:05.901021 21946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:05.902370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21879
I20260927 05:18:05.902555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:05.903116 388 external_mini_cluster.cc:1428] Setting key a592e5921fb8e2284aeeaf58708a016d
I20260927 05:18:05.914399 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb: Bootstrap starting.
I20260927 05:18:05.915841 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:05.904863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:48119 (local address 127.0.0.1:38699)
0927 05:18:05.905611 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:05.905622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:05.905661 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:18:05.907279 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:05.907300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:05.907685 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:05.907998 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:05.908555 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:05.909776 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:05.911684 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:05.912342 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:05.912460 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:05.913692 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:05.913699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:05.913711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:05.914206 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:05.914275 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:05.914304 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:05.914420 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:05.914735 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 05:18:05.914908 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":344,"threads_started":1}
I20260927 05:18:05.917426 21946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:05.918516 21946 log.cc:824] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:05.919456 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:05.921497 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb: No bootstrap required, opened a new log
I20260927 05:18:05.930743 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:05.924417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:37803 (local address 127.0.0.1:38699)
0927 05:18:05.924591 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:05.924597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:05.924994 (+ 397us) server_negotiation.cc:407] Connection header received
0927 05:18:05.925115 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:05.925121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:05.925197 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:05.925290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:05.925757 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:05.926870 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:05.928199 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:05.928909 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:05.929052 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:05.930024 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:05.930030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:05.930035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:05.930053 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:05.930109 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:05.930127 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:05.930231 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:05.930376 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 05:18:05.930423 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:18:05.931059 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260927 05:18:05.931473 21946 raft_consensus.cc:374] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:05.931604 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f241d5e689514a1c83a43c51a41e30eb, State: Initialized, Role: FOLLOWER
I20260927 05:18:05.932286 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260927 05:18:05.932538 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:05.932639 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:05.932758 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:05.934912 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260927 05:18:05.935482 21946 leader_election.cc:302] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f241d5e689514a1c83a43c51a41e30eb; no voters:
I20260927 05:18:05.936026 21946 leader_election.cc:288] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:05.936159 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:05.936854 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [term 1 LEADER]: Becoming Leader. State: Replica: f241d5e689514a1c83a43c51a41e30eb, State: Running, Role: LEADER
I20260927 05:18:05.937368 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260927 05:18:05.937958 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:05.940104 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f241d5e689514a1c83a43c51a41e30eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260927 05:18:05.940541 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:05.941098 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f241d5e689514a1c83a43c51a41e30eb. Latest consensus state: current_term: 1 leader_uuid: "f241d5e689514a1c83a43c51a41e30eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241d5e689514a1c83a43c51a41e30eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260927 05:18:05.941533 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:05.942786 21963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:05.948715 21963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:05.957293 21963 catalog_manager.cc:1347] Generated new cluster ID: e7fae965d54545f781e95f3e22067933
I20260927 05:18:05.957408 21963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:05.964876 21963 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:05.966008 21963 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:05.972720 21963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f241d5e689514a1c83a43c51a41e30eb: Generated new TSK 0
I20260927 05:18:05.973402 21963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:05.987084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21879
W20260927 05:18:05.988257 21941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:48119
W20260927 05:18:05.988951 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21879: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:05.989084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21879
W20260927 05:18:06.006072 21864 connection.cc:535] client connection to 127.0.0.1:38699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38699): Connection reset by peer (error 104)
2026-09-27T05:18:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1836 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:06 dist-test-slave-1t6p krb5kdc[21977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:06 dist-test-slave-1t6p krb5kdc[21977](info): set up 2 sockets
Sep 27 05:18:06 dist-test-slave-1t6p krb5kdc[21977](info): commencing operation
krb5kdc: starting...
W20260927 05:18:07.079342 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:07 dist-test-slave-1t6p krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:18:07.307854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:07.623746 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:07.624048 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:07.624198 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:07.631182 21990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:07.631304 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:07.631402 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:07.631505 21990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:07.631584 21990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:07.640785 21990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:07.641938 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:07.643186 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:07.654290 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:07.654253 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:07.654337 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:07.655077 21990 server_base.cc:1034] running on GCE node
I20260927 05:18:07.655766 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:07.657012 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:07.658234 21990 hybrid_clock.cc:648] HybridClock initialized: now 1790486287658200 us; error 43 us; skew 500 ppm
I20260927 05:18:07.658655 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:07 dist-test-slave-1t6p krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:07.668423 21990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:18:07.669731 21990 webserver.cc:460] Webserver started at http://127.0.0.1:34077/ using document root <none> and password file <none>
I20260927 05:18:07.670354 21990 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:07.670450 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:07.670686 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:07.673514 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "2c7860bf57a94acb8e2e8adc70eeaccc"
format_stamp: "Formatted at 2026-09-27 05:18:07 on dist-test-slave-1t6p"
server_key: "df5f4a4e985ce98bbaa0105b8b242a2e"
server_key_iv: "224eac7e3c08ebe9aa171c2698fb13eb"
server_key_version: "encryptionkey@0"
I20260927 05:18:07.674206 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c7860bf57a94acb8e2e8adc70eeaccc"
format_stamp: "Formatted at 2026-09-27 05:18:07 on dist-test-slave-1t6p"
server_key: "df5f4a4e985ce98bbaa0105b8b242a2e"
server_key_iv: "224eac7e3c08ebe9aa171c2698fb13eb"
server_key_version: "encryptionkey@0"
I20260927 05:18:07.678956 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 05:18:07.682092 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:07.683372 21990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:18:07.683501 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "2c7860bf57a94acb8e2e8adc70eeaccc"
format_stamp: "Formatted at 2026-09-27 05:18:07 on dist-test-slave-1t6p"
server_key: "df5f4a4e985ce98bbaa0105b8b242a2e"
server_key_iv: "224eac7e3c08ebe9aa171c2698fb13eb"
server_key_version: "encryptionkey@0"
I20260927 05:18:07.683647 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:07.688722 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:07.692287 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:07.692507 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:07.710295 21990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33605
I20260927 05:18:07.710369 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33605 every 8 connection(s)
I20260927 05:18:07.711647 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:07.715157 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:07.717095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 21990
I20260927 05:18:07.717260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:07.717686 388 external_mini_cluster.cc:1428] Setting key f5756064b276c3a1908a3a71a10e0004
I20260927 05:18:07.724838 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:07.719318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:55195 (local address 127.0.0.1:33605)
0927 05:18:07.720064 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:07.720078 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:07.720120 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:18:07.721737 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:07.721758 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:07.722064 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:07.722369 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:07.723384 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:07.723394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:07.723406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:07.723419 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:07.723477 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:07.723500 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:07.723656 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:07.723758 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 05:18:07.723988 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":353,"threads_started":1}
I20260927 05:18:07.726595 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc: Bootstrap starting.
I20260927 05:18:07.728271 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:07.729696 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:07.730626 22057 log.cc:824] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:07.733439 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc: No bootstrap required, opened a new log
I20260927 05:18:07.734758 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:07.732511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:53099 (local address 127.0.0.1:33605)
0927 05:18:07.732693 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:07.732699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:07.732993 (+ 294us) server_negotiation.cc:407] Connection header received
0927 05:18:07.733172 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:07.733176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:07.733215 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:07.733288 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:07.734133 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:07.734137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:07.734139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:07.734162 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:07.734195 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:07.734208 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:07.734294 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:07.734421 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 05:18:07.734468 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:18:07.740731 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } }
I20260927 05:18:07.741084 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:07.741182 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7860bf57a94acb8e2e8adc70eeaccc, State: Initialized, Role: FOLLOWER
I20260927 05:18:07.741734 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } }
I20260927 05:18:07.741986 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:07.742130 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:07.742301 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:07.744377 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } }
I20260927 05:18:07.744840 22057 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7860bf57a94acb8e2e8adc70eeaccc; no voters:
I20260927 05:18:07.745379 22057 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:07.745505 22067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:07.746201 22067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [term 1 LEADER]: Becoming Leader. State: Replica: 2c7860bf57a94acb8e2e8adc70eeaccc, State: Running, Role: LEADER
I20260927 05:18:07.746841 22067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } }
I20260927 05:18:07.747507 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:07.749683 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } } }
I20260927 05:18:07.750141 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:07.750326 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7860bf57a94acb8e2e8adc70eeaccc. Latest consensus state: current_term: 1 leader_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7860bf57a94acb8e2e8adc70eeaccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33605 } } }
I20260927 05:18:07.750681 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:07.752787 22075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:07.757529 22075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:07.765522 22075 catalog_manager.cc:1347] Generated new cluster ID: 7e41365ec5534d13a3af3f032c23d3b9
I20260927 05:18:07.765635 22075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:07.807968 22075 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:07.808451 22075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:07.817747 22075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c7860bf57a94acb8e2e8adc70eeaccc: Generated new TSK 0
I20260927 05:18:07.818655 22075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:07.833335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21990
W20260927 05:18:07.834337 22052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:55195
W20260927 05:18:07.834985 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 21990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:07.835168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 21990
2026-09-27T05:18:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:07 dist-test-slave-1t6p krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:07 dist-test-slave-1t6p krb5kdc[22088](info): set up 2 sockets
Sep 27 05:18:07 dist-test-slave-1t6p krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260927 05:18:08.923313 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:09.155884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39879
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:09.468626 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:09.468887 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:09.469033 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:09.475992 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:09.476090 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:09.476212 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:09.476308 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:09.476384 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:09.485790 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:09.486989 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:09.488265 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:09.499683 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:09.499575 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:09.499650 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:09.500408 22101 server_base.cc:1034] running on GCE node
I20260927 05:18:09.501175 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:09.502476 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:09.503688 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790486289503659 us; error 50 us; skew 500 ppm
I20260927 05:18:09.504012 22101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:09.507073 22101 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:09.508484 22101 webserver.cc:460] Webserver started at http://127.0.97.62:44315/ using document root <none> and password file <none>
I20260927 05:18:09.509150 22101 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:09.509286 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:09.509567 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:09.512413 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "42107ba68bc84ee49361e456137cb104"
format_stamp: "Formatted at 2026-09-27 05:18:09 on dist-test-slave-1t6p"
server_key: "b9b1d6148db4fcd10edfc6be19b4a783"
server_key_iv: "145f738c5e767fae2b04e34fb63ee004"
server_key_version: "encryptionkey@0"
I20260927 05:18:09.513175 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "42107ba68bc84ee49361e456137cb104"
format_stamp: "Formatted at 2026-09-27 05:18:09 on dist-test-slave-1t6p"
server_key: "b9b1d6148db4fcd10edfc6be19b4a783"
server_key_iv: "145f738c5e767fae2b04e34fb63ee004"
server_key_version: "encryptionkey@0"
I20260927 05:18:09.518020 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 05:18:09.521245 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:09.522532 22101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:18:09.522704 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "42107ba68bc84ee49361e456137cb104"
format_stamp: "Formatted at 2026-09-27 05:18:09 on dist-test-slave-1t6p"
server_key: "b9b1d6148db4fcd10edfc6be19b4a783"
server_key_iv: "145f738c5e767fae2b04e34fb63ee004"
server_key_version: "encryptionkey@0"
I20260927 05:18:09.522886 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:09.543689 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:09.547413 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:09.547681 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:09.565819 22101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39879
I20260927 05:18:09.565853 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39879 every 8 connection(s)
I20260927 05:18:09.567126 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:09.570497 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:09.574946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22101
I20260927 05:18:09.575163 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:09.575747 388 external_mini_cluster.cc:1428] Setting key 939bfc3ea79ed6fb24f5ec94339e8da9
I20260927 05:18:09.581497 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104: Bootstrap starting.
I20260927 05:18:09.584371 22168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:09.585314 22168 log.cc:824] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:09.587880 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104: No bootstrap required, opened a new log
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:09.596702 22168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } }
I20260927 05:18:09.597044 22168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:09.597177 22168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42107ba68bc84ee49361e456137cb104, State: Initialized, Role: FOLLOWER
I20260927 05:18:09.597600 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:09.577482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55345 (local address 127.0.97.62:39879)
0927 05:18:09.578222 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:09.578233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:09.578273 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:18:09.580308 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:09.580330 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:09.580714 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:09.581063 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:09.582408 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:09.583788 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:09.585097 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:09.585793 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:09.585986 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:09.590358 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:09.590368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:09.590381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:09.590417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:09.593515 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:09.594354 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:09.594360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:09.594365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:09.594527 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:09.595040 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:09.595049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:09.595051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:09.595527 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:09.595756 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:09.596236 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 05:18:09.596509 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260927 05:18:09.597841 22168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } }
I20260927 05:18:09.598101 22168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:09.598260 22168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:09.598441 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:09.600939 22168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } }
I20260927 05:18:09.601296 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:09.601464 22168 leader_election.cc:302] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42107ba68bc84ee49361e456137cb104; no voters:
I20260927 05:18:09.602017 22168 leader_election.cc:288] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:09.602118 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:09.603996 22168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:09.609140 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [term 1 LEADER]: Becoming Leader. State: Replica: 42107ba68bc84ee49361e456137cb104, State: Running, Role: LEADER
I20260927 05:18:09.609814 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } }
I20260927 05:18:09.613003 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42107ba68bc84ee49361e456137cb104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } } }
I20260927 05:18:09.613471 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:09.614092 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42107ba68bc84ee49361e456137cb104. Latest consensus state: current_term: 1 leader_uuid: "42107ba68bc84ee49361e456137cb104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42107ba68bc84ee49361e456137cb104" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39879 } } }
I20260927 05:18:09.614416 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:09.635695 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:09.637522 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:09.607491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42133 (local address 127.0.97.62:39879)
0927 05:18:09.607716 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:09.607724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:09.616844 (+ 9120us) server_negotiation.cc:407] Connection header received
0927 05:18:09.617318 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:09.617326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:09.617422 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:09.617499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:09.620197 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:09.621655 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:09.624076 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:09.624941 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:09.625065 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:09.629348 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:09.629357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:09.629365 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:09.629405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:09.633792 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:09.634863 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:09.634869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:09.634874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:09.635029 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:09.635638 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:09.635645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:09.635651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:09.636095 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:09.636340 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:09.636992 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 05:18:09.637166 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:18:09.645947 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:09.654759 22187 catalog_manager.cc:1347] Generated new cluster ID: 041db5e12dc4416ea6d1f77b5146d9cd
I20260927 05:18:09.654855 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:09.708845 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:09.710117 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:09.718058 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42107ba68bc84ee49361e456137cb104: Generated new TSK 0
I20260927 05:18:09.719069 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:09.733217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22101
I20260927 05:18:09.734395 22163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:09.827368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22101
W20260927 05:18:09.844491 22086 connection.cc:535] client connection to 127.0.97.62:39879 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:39879): Connection reset by peer (error 104)
2026-09-27T05:18:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22201](info): set up 2 sockets
Sep 27 05:18:09 dist-test-slave-1t6p krb5kdc[22201](info): commencing operation
krb5kdc: starting...
W20260927 05:18:10.915398 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:11 dist-test-slave-1t6p krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:11.146353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:11.463495 22214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:11.463814 22214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:11.463968 22214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:11.470789 22214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:11.470930 22214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:11.471048 22214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:11.471153 22214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:11.471221 22214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:11.480433 22214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:11.481648 22214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:11.482901 22214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:11.494028 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:11.494071 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:11.494112 22222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:11.495201 22214 server_base.cc:1034] running on GCE node
I20260927 05:18:11.495952 22214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:11.497246 22214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:11.498458 22214 hybrid_clock.cc:648] HybridClock initialized: now 1790486291498414 us; error 40 us; skew 500 ppm
I20260927 05:18:11.498783 22214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:11 dist-test-slave-1t6p krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:11.501724 22214 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:11.503110 22214 webserver.cc:460] Webserver started at http://127.0.97.62:37143/ using document root <none> and password file <none>
I20260927 05:18:11.503739 22214 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:11.503873 22214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:11.504165 22214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:11.506923 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "70edb8c42d664af39263f56f21c78e29"
format_stamp: "Formatted at 2026-09-27 05:18:11 on dist-test-slave-1t6p"
server_key: "de26c2a18c7dba044e4625368f206e91"
server_key_iv: "1ad5b19e2b76e6452a02150838859a02"
server_key_version: "encryptionkey@0"
I20260927 05:18:11.507699 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "70edb8c42d664af39263f56f21c78e29"
format_stamp: "Formatted at 2026-09-27 05:18:11 on dist-test-slave-1t6p"
server_key: "de26c2a18c7dba044e4625368f206e91"
server_key_iv: "1ad5b19e2b76e6452a02150838859a02"
server_key_version: "encryptionkey@0"
I20260927 05:18:11.512610 22214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:18:11.515785 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:11.516971 22214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 05:18:11.517149 22214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "70edb8c42d664af39263f56f21c78e29"
format_stamp: "Formatted at 2026-09-27 05:18:11 on dist-test-slave-1t6p"
server_key: "de26c2a18c7dba044e4625368f206e91"
server_key_iv: "1ad5b19e2b76e6452a02150838859a02"
server_key_version: "encryptionkey@0"
I20260927 05:18:11.517328 22214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:11.535725 22214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:11.539427 22214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:11.539706 22214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:11.557933 22214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40259
I20260927 05:18:11.557981 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40259 every 8 connection(s)
I20260927 05:18:11.559269 22214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:11.562747 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:11.566071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22214
I20260927 05:18:11.566267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:11.566766 388 external_mini_cluster.cc:1428] Setting key f40ce88ba657902e646c0f1ca50a44bb
I20260927 05:18:11.574721 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29: Bootstrap starting.
I20260927 05:18:11.578187 22281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:11.579411 22281 log.cc:824] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:11.580201 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:11.568467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32785 (local address 127.0.97.62:40259)
0927 05:18:11.569268 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:11.569280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:11.569321 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:18:11.571048 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:11.571082 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:11.571493 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:11.571835 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:11.572385 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:11.573798 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:11.575638 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:11.576301 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:11.576448 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:11.577694 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:11.577701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:11.577714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:11.578213 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:11.578345 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:11.578376 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:11.578503 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:11.578883 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 05:18:11.579098 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":391,"threads_started":1}
I20260927 05:18:11.582095 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29: No bootstrap required, opened a new log
I20260927 05:18:11.584038 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:11.591952 22281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260927 05:18:11.592319 22281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:11.592417 22281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70edb8c42d664af39263f56f21c78e29, State: Initialized, Role: FOLLOWER
I20260927 05:18:11.592984 22281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260927 05:18:11.593228 22281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:11.593369 22281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:11.593541 22281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:11.595726 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:11.589042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48873 (local address 127.0.97.62:40259)
0927 05:18:11.589246 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:11.589252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:11.589804 (+ 552us) server_negotiation.cc:407] Connection header received
0927 05:18:11.589931 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:11.589937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:11.590016 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:11.590130 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:11.590627 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:11.591730 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:11.593173 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:11.593874 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:11.594013 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:11.595026 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:11.595033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:11.595037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:11.595060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:11.595122 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:11.595142 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:11.595259 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:11.595364 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 05:18:11.595422 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:18:11.596657 22281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260927 05:18:11.597337 22281 leader_election.cc:302] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70edb8c42d664af39263f56f21c78e29; no voters:
I20260927 05:18:11.597887 22281 leader_election.cc:288] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:11.598009 22291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:11.598656 22291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [term 1 LEADER]: Becoming Leader. State: Replica: 70edb8c42d664af39263f56f21c78e29, State: Running, Role: LEADER
I20260927 05:18:11.599197 22291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260927 05:18:11.599848 22281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:11.602042 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70edb8c42d664af39263f56f21c78e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } }
I20260927 05:18:11.602470 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:11.606330 22297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:11.606226 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70edb8c42d664af39263f56f21c78e29. Latest consensus state: current_term: 1 leader_uuid: "70edb8c42d664af39263f56f21c78e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70edb8c42d664af39263f56f21c78e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } }
I20260927 05:18:11.606523 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:11.611104 22297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:11.620600 22297 catalog_manager.cc:1347] Generated new cluster ID: 9c5524a040344161a0c2c04f7c3b9d5b
I20260927 05:18:11.620713 22297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:11.630141 22297 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:11.631599 22297 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:11.640275 22297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70edb8c42d664af39263f56f21c78e29: Generated new TSK 0
I20260927 05:18:11.641007 22297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:11.652499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22214
W20260927 05:18:11.654026 22276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32785
W20260927 05:18:11.654707 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:11.654850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22214
W20260927 05:18:11.673318 22199 connection.cc:535] client connection to 127.0.97.62:40259 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40259): Connection reset by peer (error 104)
2026-09-27T05:18:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:11 dist-test-slave-1t6p krb5kdc[22312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:11 dist-test-slave-1t6p krb5kdc[22312](info): set up 2 sockets
Sep 27 05:18:11 dist-test-slave-1t6p krb5kdc[22312](info): commencing operation
krb5kdc: starting...
W20260927 05:18:12.743551 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:12 dist-test-slave-1t6p krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:12.975955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:13.292980 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:13.293315 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:13.293473 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:13.300585 22325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:13.300734 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:13.300834 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:13.300935 22325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:13.301008 22325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:13.310386 22325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:13.311609 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:13.312976 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:13.324352 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:13.324316 22331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:13.324390 22333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:13.325367 22325 server_base.cc:1034] running on GCE node
I20260927 05:18:13.326130 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:13.327471 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:13.328685 22325 hybrid_clock.cc:648] HybridClock initialized: now 1790486293328649 us; error 30 us; skew 500 ppm
I20260927 05:18:13.329010 22325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:13 dist-test-slave-1t6p krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:13.335572 22325 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:13.336998 22325 webserver.cc:460] Webserver started at http://127.0.97.62:38857/ using document root <none> and password file <none>
I20260927 05:18:13.337651 22325 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:13.337791 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:13.338100 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:13.340999 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "5d0c2390f9a047cab6f3f46ffeb88029"
format_stamp: "Formatted at 2026-09-27 05:18:13 on dist-test-slave-1t6p"
server_key: "699d32ddb92b8ed698a04cae33b25bed"
server_key_iv: "8877c7da32d88a8750264828a26a46e6"
server_key_version: "encryptionkey@0"
I20260927 05:18:13.341724 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d0c2390f9a047cab6f3f46ffeb88029"
format_stamp: "Formatted at 2026-09-27 05:18:13 on dist-test-slave-1t6p"
server_key: "699d32ddb92b8ed698a04cae33b25bed"
server_key_iv: "8877c7da32d88a8750264828a26a46e6"
server_key_version: "encryptionkey@0"
I20260927 05:18:13.346724 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:18:13.349893 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:13.351241 22325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:18:13.351420 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "5d0c2390f9a047cab6f3f46ffeb88029"
format_stamp: "Formatted at 2026-09-27 05:18:13 on dist-test-slave-1t6p"
server_key: "699d32ddb92b8ed698a04cae33b25bed"
server_key_iv: "8877c7da32d88a8750264828a26a46e6"
server_key_version: "encryptionkey@0"
I20260927 05:18:13.351612 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:13.356997 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:13.360805 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:13.361088 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:13.379052 22325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41277
I20260927 05:18:13.379133 22391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41277 every 8 connection(s)
I20260927 05:18:13.380426 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:13.384027 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:13.386457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22325
I20260927 05:18:13.386628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:13.387212 388 external_mini_cluster.cc:1428] Setting key 43b718f79301a4fcb28a6684199871c7
I20260927 05:18:13.394886 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:13.388949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54747 (local address 127.0.97.62:41277)
0927 05:18:13.389685 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:13.389699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:13.389741 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:18:13.391505 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:13.391528 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:13.391896 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:13.392273 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:13.393283 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:13.393289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:13.393301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:13.393315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:13.393374 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:13.393398 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:13.393650 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:13.393806 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 05:18:13.393972 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":307,"threads_started":1}
I20260927 05:18:13.395972 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029: Bootstrap starting.
I20260927 05:18:13.398298 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:13.399151 22392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:13.400091 22392 log.cc:824] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:13.402881 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029: No bootstrap required, opened a new log
I20260927 05:18:13.404829 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:13.402345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53449 (local address 127.0.97.62:41277)
0927 05:18:13.402508 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:13.402516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:13.402993 (+ 477us) server_negotiation.cc:407] Connection header received
0927 05:18:13.403170 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:13.403175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:13.403227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:13.403354 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:13.404198 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:13.404203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:13.404207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:13.404224 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:13.404266 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:13.404288 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:13.404415 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:13.404468 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 05:18:13.404517 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:18:13.410395 22392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } }
I20260927 05:18:13.410761 22392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:13.410869 22392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0c2390f9a047cab6f3f46ffeb88029, State: Initialized, Role: FOLLOWER
I20260927 05:18:13.411491 22392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } }
I20260927 05:18:13.411777 22392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:13.411917 22392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:13.412077 22392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:13.414163 22392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } }
I20260927 05:18:13.414629 22392 leader_election.cc:302] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d0c2390f9a047cab6f3f46ffeb88029; no voters:
I20260927 05:18:13.415199 22392 leader_election.cc:288] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:13.415328 22402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:13.415978 22402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [term 1 LEADER]: Becoming Leader. State: Replica: 5d0c2390f9a047cab6f3f46ffeb88029, State: Running, Role: LEADER
I20260927 05:18:13.416539 22402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } }
I20260927 05:18:13.417097 22392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:13.419340 22404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d0c2390f9a047cab6f3f46ffeb88029. Latest consensus state: current_term: 1 leader_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } } }
I20260927 05:18:13.419934 22404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:13.419380 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0c2390f9a047cab6f3f46ffeb88029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41277 } } }
I20260927 05:18:13.421366 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:13.421425 22410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:13.425608 22410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:13.433387 22410 catalog_manager.cc:1347] Generated new cluster ID: 6718c60d0a3b402495ced900f7bbcc17
I20260927 05:18:13.433480 22410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:13.446645 22410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:13.447067 22410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:13.455978 22410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d0c2390f9a047cab6f3f46ffeb88029: Generated new TSK 0
I20260927 05:18:13.456845 22410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:13.464605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22325
W20260927 05:18:13.465580 22387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54747
W20260927 05:18:13.466180 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:13.466321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22325
2026-09-27T05:18:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:13 dist-test-slave-1t6p krb5kdc[22423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:13 dist-test-slave-1t6p krb5kdc[22423](info): set up 2 sockets
Sep 27 05:18:13 dist-test-slave-1t6p krb5kdc[22423](info): commencing operation
krb5kdc: starting...
W20260927 05:18:14.559322 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:14 dist-test-slave-1t6p krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:14.785553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41329
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:15.107645 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:15.107936 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:15.108080 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:15.115072 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:15.115203 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:15.115291 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:15.115388 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:15.115468 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:15.124956 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:15.126204 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:15.127501 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:15.138919 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:15.138954 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:15.139410 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:15.140074 22436 server_base.cc:1034] running on GCE node
I20260927 05:18:15.140762 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:15.142042 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:15.143257 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790486295143202 us; error 50 us; skew 500 ppm
I20260927 05:18:15.143565 22436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:15 dist-test-slave-1t6p krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:15.146528 22436 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:15.147886 22436 webserver.cc:460] Webserver started at http://127.0.97.62:36501/ using document root <none> and password file <none>
I20260927 05:18:15.148510 22436 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:15.148645 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:15.148926 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:15.151762 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "73799528209341afa955cc243948ce36"
format_stamp: "Formatted at 2026-09-27 05:18:15 on dist-test-slave-1t6p"
server_key: "4c5e477e6af68c96e84adf2bfed6e244"
server_key_iv: "38bd7b620a81212f53658d762df97a05"
server_key_version: "encryptionkey@0"
I20260927 05:18:15.152483 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "73799528209341afa955cc243948ce36"
format_stamp: "Formatted at 2026-09-27 05:18:15 on dist-test-slave-1t6p"
server_key: "4c5e477e6af68c96e84adf2bfed6e244"
server_key_iv: "38bd7b620a81212f53658d762df97a05"
server_key_version: "encryptionkey@0"
I20260927 05:18:15.157196 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:18:15.160290 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:15.161481 22436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:18:15.161651 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "73799528209341afa955cc243948ce36"
format_stamp: "Formatted at 2026-09-27 05:18:15 on dist-test-slave-1t6p"
server_key: "4c5e477e6af68c96e84adf2bfed6e244"
server_key_iv: "38bd7b620a81212f53658d762df97a05"
server_key_version: "encryptionkey@0"
I20260927 05:18:15.161830 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:15.191396 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:15.195555 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:15.195830 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:15.214054 22436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41329
I20260927 05:18:15.214093 22502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41329 every 8 connection(s)
I20260927 05:18:15.215355 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:15.218953 22503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:15.220378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22436
I20260927 05:18:15.220570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:15.221107 388 external_mini_cluster.cc:1428] Setting key 66746d5440dca6bcc260f501d4fcc86e
I20260927 05:18:15.233294 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36: Bootstrap starting.
Sep 27 05:18:15 dist-test-slave-1t6p krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:15.236755 22503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:15.237797 22503 log.cc:824] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:15.240466 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36: No bootstrap required, opened a new log
I20260927 05:18:15.243813 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:15.222776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44693 (local address 127.0.97.62:41329)
0927 05:18:15.223549 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:15.223560 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:15.223628 (+ 68us) server_negotiation.cc:407] Connection header received
0927 05:18:15.225606 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:15.225627 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:15.226000 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:15.226316 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:15.228011 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:15.229330 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:15.231177 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:15.231840 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:15.231961 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:15.236473 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:15.236481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:15.236494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:15.236532 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:15.239636 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:15.240293 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:15.240299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:15.240305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:15.240464 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:15.240836 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:15.240849 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:15.240851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:15.241429 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:15.241726 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:15.242453 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 05:18:15.242794 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":580,"thread_start_us":369,"threads_started":1}
I20260927 05:18:15.247190 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:15.248809 22503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } }
I20260927 05:18:15.249147 22503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:15.249244 22503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73799528209341afa955cc243948ce36, State: Initialized, Role: FOLLOWER
I20260927 05:18:15.249807 22503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } }
I20260927 05:18:15.250051 22503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:15.250187 22503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:15.250351 22503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:15.253125 22503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } }
I20260927 05:18:15.253582 22503 leader_election.cc:302] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73799528209341afa955cc243948ce36; no voters:
I20260927 05:18:15.254335 22503 leader_election.cc:288] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:15.254456 22513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:15.255648 22513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [term 1 LEADER]: Becoming Leader. State: Replica: 73799528209341afa955cc243948ce36, State: Running, Role: LEADER
I20260927 05:18:15.256240 22513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } }
I20260927 05:18:15.256672 22503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:15.259153 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73799528209341afa955cc243948ce36. Latest consensus state: current_term: 1 leader_uuid: "73799528209341afa955cc243948ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } } }
I20260927 05:18:15.259635 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:15.259523 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73799528209341afa955cc243948ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73799528209341afa955cc243948ce36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41329 } } }
I20260927 05:18:15.259851 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:15.268816 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:15.251265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37217 (local address 127.0.97.62:41329)
0927 05:18:15.251436 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:15.251443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:15.251934 (+ 491us) server_negotiation.cc:407] Connection header received
0927 05:18:15.252121 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:15.252126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:15.252192 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:15.252282 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:15.253654 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:15.254460 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:15.256190 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:15.256875 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:15.257000 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:15.260105 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:15.260113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:15.260124 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:15.260171 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:15.264818 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:15.265823 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:15.265832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:15.265839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:15.266033 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:15.266566 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:15.266574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:15.266580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:15.267072 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:15.267364 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:15.268035 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 05:18:15.268328 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:18:15.270563 22519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:15.275915 22519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:15.284417 22519 catalog_manager.cc:1347] Generated new cluster ID: 47b34e11d9c8493cb9da37968cbab13c
I20260927 05:18:15.284524 22519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:15.297835 22519 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:15.299099 22519 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:15.311479 22519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73799528209341afa955cc243948ce36: Generated new TSK 0
I20260927 05:18:15.312445 22519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:15.322017 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22436
I20260927 05:18:15.323434 22498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:15.412551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22436
2026-09-27T05:18:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:15 dist-test-slave-1t6p krb5kdc[22535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:15 dist-test-slave-1t6p krb5kdc[22535](info): set up 2 sockets
Sep 27 05:18:15 dist-test-slave-1t6p krb5kdc[22535](info): commencing operation
krb5kdc: starting...
W20260927 05:18:16.515462 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:16 dist-test-slave-1t6p krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:16.758625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43213
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:17.073220 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:17.073468 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:17.073561 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:17.080462 22548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:17.080562 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:17.080608 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:17.080667 22548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:17.080716 22548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:17.089813 22548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:17.090886 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:17.092105 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:17.103243 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:17.103268 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:17.103542 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:17.104606 22548 server_base.cc:1034] running on GCE node
I20260927 05:18:17.105415 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:17.106760 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:17.107988 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790486297107906 us; error 51 us; skew 500 ppm
I20260927 05:18:17.108330 22548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:17 dist-test-slave-1t6p krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:17.111325 22548 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:17.112596 22548 webserver.cc:460] Webserver started at http://127.0.97.62:38189/ using document root <none> and password file <none>
I20260927 05:18:17.113189 22548 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:17.113283 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:17.113513 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:17.116384 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "123931524a5e4e0f8f2ada1c9c084228"
format_stamp: "Formatted at 2026-09-27 05:18:17 on dist-test-slave-1t6p"
server_key: "847cc1d6892fac9ad7484f1b458b5ecc"
server_key_iv: "5e220fa1d2fc6acab5b4e24ca2a09780"
server_key_version: "encryptionkey@0"
I20260927 05:18:17.117066 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "123931524a5e4e0f8f2ada1c9c084228"
format_stamp: "Formatted at 2026-09-27 05:18:17 on dist-test-slave-1t6p"
server_key: "847cc1d6892fac9ad7484f1b458b5ecc"
server_key_iv: "5e220fa1d2fc6acab5b4e24ca2a09780"
server_key_version: "encryptionkey@0"
I20260927 05:18:17.121990 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:18:17.125195 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:17.126423 22548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 05:18:17.126611 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "123931524a5e4e0f8f2ada1c9c084228"
format_stamp: "Formatted at 2026-09-27 05:18:17 on dist-test-slave-1t6p"
server_key: "847cc1d6892fac9ad7484f1b458b5ecc"
server_key_iv: "5e220fa1d2fc6acab5b4e24ca2a09780"
server_key_version: "encryptionkey@0"
I20260927 05:18:17.126770 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:17.146782 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:17.150796 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:17.151077 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:17.168573 22548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43213
I20260927 05:18:17.168574 22614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43213 every 8 connection(s)
I20260927 05:18:17.169903 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:17.173460 22615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:17.178982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22548
I20260927 05:18:17.179229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:17.179669 388 external_mini_cluster.cc:1428] Setting key ae56ebfca30586b0fd6265316fa174e6
I20260927 05:18:17.184139 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: Bootstrap starting.
I20260927 05:18:17.187527 22615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:17.188730 22615 log.cc:824] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:17.191779 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: No bootstrap required, opened a new log
I20260927 05:18:17.192890 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:17.181326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39607 (local address 127.0.97.62:43213)
0927 05:18:17.182208 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:17.182219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:17.182261 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:18:17.183974 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:17.184004 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:17.184415 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:17.184727 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:17.185295 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:17.186651 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:17.188559 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:17.189440 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:17.189643 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:17.190670 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:17.190678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:17.190691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:17.191325 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:17.191398 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:17.191422 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:17.191518 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:17.191740 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 05:18:17.191985 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":451,"threads_started":1}
I20260927 05:18:17.196589 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:17.199930 22615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } }
I20260927 05:18:17.200284 22615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:17.200394 22615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123931524a5e4e0f8f2ada1c9c084228, State: Initialized, Role: FOLLOWER
I20260927 05:18:17.201094 22615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } }
I20260927 05:18:17.201382 22615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:17.201539 22615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:17.201735 22615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:17.204003 22615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } }
I20260927 05:18:17.204499 22615 leader_election.cc:302] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123931524a5e4e0f8f2ada1c9c084228; no voters:
I20260927 05:18:17.205117 22615 leader_election.cc:288] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:17.205245 22625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:17.206941 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:17.200685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51925 (local address 127.0.97.62:43213)
0927 05:18:17.200904 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:17.200913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:17.201351 (+ 438us) server_negotiation.cc:407] Connection header received
0927 05:18:17.201545 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:17.201550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:17.201619 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:17.201741 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:17.202242 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:17.202964 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:17.204291 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:17.204798 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:17.204960 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:17.206125 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:17.206131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:17.206135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:17.206154 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:17.206220 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:17.206240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:17.206301 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:17.206526 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 05:18:17.206612 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:18:17.207589 22615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:17.212306 22625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [term 1 LEADER]: Becoming Leader. State: Replica: 123931524a5e4e0f8f2ada1c9c084228, State: Running, Role: LEADER
I20260927 05:18:17.212954 22625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } }
I20260927 05:18:17.216161 22626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123931524a5e4e0f8f2ada1c9c084228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } } }
I20260927 05:18:17.216737 22626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:17.216171 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123931524a5e4e0f8f2ada1c9c084228. Latest consensus state: current_term: 1 leader_uuid: "123931524a5e4e0f8f2ada1c9c084228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123931524a5e4e0f8f2ada1c9c084228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43213 } } }
I20260927 05:18:17.217135 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228 [sys.catalog]: This master's current role is: LEADER
W20260927 05:18:17.221644 22639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:17.221877 22639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:17.223567 22640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:17.225126 22640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:17.232797 22640 catalog_manager.cc:1347] Generated new cluster ID: e67c745299f14c02a460da968012a359
I20260927 05:18:17.232897 22640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:17.239934 22640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:17.241593 22640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:17.261032 22640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123931524a5e4e0f8f2ada1c9c084228: Generated new TSK 0
I20260927 05:18:17.262009 22640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:17.268252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22548
W20260927 05:18:17.269282 22610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39607
W20260927 05:18:17.269984 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:17.270126 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22548
2026-09-27T05:18:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:17 dist-test-slave-1t6p krb5kdc[22646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:17 dist-test-slave-1t6p krb5kdc[22646](info): set up 2 sockets
Sep 27 05:18:17 dist-test-slave-1t6p krb5kdc[22646](info): commencing operation
krb5kdc: starting...
W20260927 05:18:18.351577 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:18 dist-test-slave-1t6p krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:18.577461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:18.897141 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:18.897413 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:18.897593 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:18.904690 22659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:18.904795 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:18.904910 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:18.905021 22659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:18.905088 22659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:18.914597 22659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:18.915817 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:18.917058 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:18.928678 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:18.928709 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:18.928607 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:18.929452 22659 server_base.cc:1034] running on GCE node
I20260927 05:18:18.930222 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:18.931624 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:18.932858 22659 hybrid_clock.cc:648] HybridClock initialized: now 1790486298932812 us; error 40 us; skew 500 ppm
I20260927 05:18:18.933208 22659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:18 dist-test-slave-1t6p krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:18.942027 22659 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:18.943619 22659 webserver.cc:460] Webserver started at http://127.0.97.62:40593/ using document root <none> and password file <none>
I20260927 05:18:18.944280 22659 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:18.944424 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:18.944730 22659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:18.947624 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "5226c45c87624442a32f09977fde23d2"
format_stamp: "Formatted at 2026-09-27 05:18:18 on dist-test-slave-1t6p"
server_key: "73f15b91f49701a5952d8e93a2787abd"
server_key_iv: "5ee4c971cb22bc8a5869bb09359c9cf0"
server_key_version: "encryptionkey@0"
I20260927 05:18:18.948364 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "5226c45c87624442a32f09977fde23d2"
format_stamp: "Formatted at 2026-09-27 05:18:18 on dist-test-slave-1t6p"
server_key: "73f15b91f49701a5952d8e93a2787abd"
server_key_iv: "5ee4c971cb22bc8a5869bb09359c9cf0"
server_key_version: "encryptionkey@0"
I20260927 05:18:18.953406 22659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:18:18.956878 22674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:18.958288 22659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:18:18.958503 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "5226c45c87624442a32f09977fde23d2"
format_stamp: "Formatted at 2026-09-27 05:18:18 on dist-test-slave-1t6p"
server_key: "73f15b91f49701a5952d8e93a2787abd"
server_key_iv: "5ee4c971cb22bc8a5869bb09359c9cf0"
server_key_version: "encryptionkey@0"
I20260927 05:18:18.958705 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:18.964375 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:18.968305 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:18.968592 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:18.986358 22659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45219
I20260927 05:18:18.986430 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45219 every 8 connection(s)
I20260927 05:18:18.987689 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:18.988963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22659
I20260927 05:18:18.989151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:18.989645 388 external_mini_cluster.cc:1428] Setting key 59db71bbdebd2b8fbf07a4b988525097
I20260927 05:18:18.991998 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:18.997025 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:18.991384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.0.97.62:45219)
0927 05:18:18.992176 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:18.992188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:18.992225 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:18:18.993791 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:18.993811 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:18.994121 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:18.994493 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:18.995542 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:18.995548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:18.995559 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:18.995572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:18.995635 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:18.995652 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:18.995895 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:18.996042 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:18:18.996196 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":329,"threads_started":1}
I20260927 05:18:19.000420 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:19.002633 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2: Bootstrap starting.
I20260927 05:18:19.005836 22726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:19.007211 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:19.004771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51379 (local address 127.0.97.62:45219)
0927 05:18:19.004932 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:19.004939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:19.005403 (+ 464us) server_negotiation.cc:407] Connection header received
0927 05:18:19.005533 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:19.005540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:19.005601 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:19.005705 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:19.006562 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:19.006567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:19.006572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:18:19.006589 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:19.006631 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:18:19.006649 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:19.006765 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:19.006842 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 05:18:19.006895 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:18:19.007187 22726 log.cc:824] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:19.009814 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2: No bootstrap required, opened a new log
I20260927 05:18:19.016824 22726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260927 05:18:19.017170 22726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:18:19.017234 22726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5226c45c87624442a32f09977fde23d2, State: Initialized, Role: FOLLOWER
I20260927 05:18:19.017736 22726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260927 05:18:19.017935 22726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:18:19.018028 22726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:18:19.018144 22726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:19.020498 22726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260927 05:18:19.020960 22726 leader_election.cc:302] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5226c45c87624442a32f09977fde23d2; no voters:
I20260927 05:18:19.021471 22726 leader_election.cc:288] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:18:19.021625 22736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:19.022269 22736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5226c45c87624442a32f09977fde23d2, State: Running, Role: LEADER
I20260927 05:18:19.022820 22736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260927 05:18:19.023571 22726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:19.025899 22738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5226c45c87624442a32f09977fde23d2. Latest consensus state: current_term: 1 leader_uuid: "5226c45c87624442a32f09977fde23d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260927 05:18:19.026273 22738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:19.028046 22742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:19.027917 22737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5226c45c87624442a32f09977fde23d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5226c45c87624442a32f09977fde23d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260927 05:18:19.028219 22737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:19.034721 22742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:19.042879 22742 catalog_manager.cc:1347] Generated new cluster ID: d912a5fcbb804faaa3167804d2475eb6
I20260927 05:18:19.042985 22742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:19.082979 22742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:19.083416 22742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:19.091249 22742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5226c45c87624442a32f09977fde23d2: Generated new TSK 0
I20260927 05:18:19.092182 22742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:18:19.113780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22659
W20260927 05:18:19.114794 22721 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60995
W20260927 05:18:19.115490 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:18:19.115634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22659
2026-09-27T05:18:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1846 ms)
[----------] 18 tests from AuthTokenIssuingTest (33870 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:19 dist-test-slave-1t6p krb5kdc[22757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:19 dist-test-slave-1t6p krb5kdc[22757](info): set up 2 sockets
Sep 27 05:18:19 dist-test-slave-1t6p krb5kdc[22757](info): commencing operation
krb5kdc: starting...
W20260927 05:18:20.243590 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:20 dist-test-slave-1t6p krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:20.484069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:20.802161 22770 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:20.802608 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:20.802744 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:20.802886 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:20.810048 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:20.810163 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:20.810286 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:20.810374 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:20.810470 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:20.820168 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:20.821429 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:20.822705 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:20.834609 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:20.834602 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:20.834635 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:20.835618 22770 server_base.cc:1034] running on GCE node
I20260927 05:18:20.836449 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:20.837915 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:20.839140 22770 hybrid_clock.cc:648] HybridClock initialized: now 1790486300839081 us; error 31 us; skew 500 ppm
I20260927 05:18:20.839478 22770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:20 dist-test-slave-1t6p krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:20.842597 22770 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:20.844075 22770 webserver.cc:460] Webserver started at http://127.0.97.62:39385/ using document root <none> and password file <none>
I20260927 05:18:20.844759 22770 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:20.844888 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:20.845208 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:20.848121 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "ef88ef50994442c39f06b81142d472b2"
format_stamp: "Formatted at 2026-09-27 05:18:20 on dist-test-slave-1t6p"
server_key: "66cd8afa20f79bc5f202047f49b0ba7b"
server_key_iv: "65ccff25e8e989d635ebe5d25ee758d5"
server_key_version: "encryptionkey@0"
I20260927 05:18:20.848881 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef88ef50994442c39f06b81142d472b2"
format_stamp: "Formatted at 2026-09-27 05:18:20 on dist-test-slave-1t6p"
server_key: "66cd8afa20f79bc5f202047f49b0ba7b"
server_key_iv: "65ccff25e8e989d635ebe5d25ee758d5"
server_key_version: "encryptionkey@0"
I20260927 05:18:20.853875 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:18:20.857358 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:20.858664 22770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:18:20.858848 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "ef88ef50994442c39f06b81142d472b2"
format_stamp: "Formatted at 2026-09-27 05:18:20 on dist-test-slave-1t6p"
server_key: "66cd8afa20f79bc5f202047f49b0ba7b"
server_key_iv: "65ccff25e8e989d635ebe5d25ee758d5"
server_key_version: "encryptionkey@0"
I20260927 05:18:20.859107 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:20.893558 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:20.897661 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:20.897929 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:20.916378 22770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33219
I20260927 05:18:20.916415 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33219 every 8 connection(s)
I20260927 05:18:20.917681 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:20.921224 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:20.925841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22770
I20260927 05:18:20.926059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:20.926882 388 external_mini_cluster.cc:1428] Setting key 4ce7a0d00addb1efd8282e55639a9051
I20260927 05:18:20.928648 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:20 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 05:18:20.956377 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:20.932404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.97.62:33219)
0927 05:18:20.937058 (+ 4654us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:20.937065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:20.937092 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:18:20.937216 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:20.937221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:20.937417 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:20.937573 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:20.939371 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:20.940561 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:20.942961 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:20.943559 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:20.943686 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:20.947989 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:20.948001 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:20.948018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:20.948074 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:20.952191 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:20.952910 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:20.952921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:20.952927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:20.953144 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:20.953535 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:20.953542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:20.953546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:20.954261 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:20.954471 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:20.955043 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 05:18:20.955447 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":4491,"spinlock_wait_cycles":14720,"thread_start_us":4317,"threads_started":1}
I20260927 05:18:20.956506 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:20.932404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.0.1:34816)
0927 05:18:20.933306 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:20.933353 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:20.936226 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:20.937681 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:20.937693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:20.938582 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:20.939150 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:20.939178 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:20.940727 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:20.940737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:20.942805 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:20.942825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:20.943809 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:20.943819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:20.944081 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:20.944828 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:20.944863 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:20.947753 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:20.952353 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:20.952363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:20.952371 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:20.952753 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:20.953256 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:20.953259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:20.953261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:20.953414 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:20.954402 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:20.954408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:20.954603 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 05:18:20.954805 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 05:18:20.955262 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":381,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 05:18:20.959288 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:20.960284 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:20.959916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45116)
0927 05:18:20.960098 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:20.960191 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:18:20.960505 22786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
I20260927 05:18:20.961983 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:20.961634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45130)
0927 05:18:20.961784 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:20.961849 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:18:20.962203 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46647: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:18:20.986793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46647
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:18:21.011521 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } attempt: 1
I20260927 05:18:21.012729 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.012282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45146)
0927 05:18:21.012496 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.012588 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:18:21.013808 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.013458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45154)
0927 05:18:21.013619 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.013685 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:18:21.014003 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46647: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
I20260927 05:18:21.082266 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } attempt: 2
I20260927 05:18:21.083532 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.082973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45156)
0927 05:18:21.083293 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.083387 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 05:18:21.084633 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.084240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45172)
0927 05:18:21.084400 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.084495 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:18:21.084816 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46647: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
I20260927 05:18:21.176057 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } attempt: 3
I20260927 05:18:21.177203 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.176678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45186)
0927 05:18:21.176912 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.177026 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:18:21.178264 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.177856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45194)
0927 05:18:21.178005 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.178093 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:18:21.178484 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46647: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
W20260927 05:18:21.306010 22844 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:21.306322 22844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:21.306452 22844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:21.306593 22844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:21.313470 22844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:21.313551 22844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:21.313651 22844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:21.313742 22844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:21.313828 22844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:21.321738 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } attempt: 4
I20260927 05:18:21.322992 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.322393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45210)
0927 05:18:21.322700 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.322831 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260927 05:18:21.323700 22844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:21.324060 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.323657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45216)
0927 05:18:21.323811 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.323907 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:18:21.324266 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46647: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46647: connect: Connection refused (error 111)
I20260927 05:18:21.324990 22844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:21.326206 22844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:21.337422 22850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:21.337474 22849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:21.337471 22852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:21.338512 22844 server_base.cc:1034] running on GCE node
I20260927 05:18:21.339275 22844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:21.340562 22844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:21.341845 22844 hybrid_clock.cc:648] HybridClock initialized: now 1790486301341759 us; error 47 us; skew 500 ppm
I20260927 05:18:21.342170 22844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:21.345386 22844 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:18:21.346704 22844 webserver.cc:460] Webserver started at http://127.0.97.61:41843/ using document root <none> and password file <none>
I20260927 05:18:21.347409 22844 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:21.347493 22844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:21.347851 22844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:21.350731 22844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data/instance:
uuid: "4e02fbd7f5694a34a54f6490d90956fe"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "16cb66a353083843aeefba4be7d58d87"
server_key_iv: "b2a687e91ed2e475492a95e004218479"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.351553 22844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal/instance:
uuid: "4e02fbd7f5694a34a54f6490d90956fe"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "16cb66a353083843aeefba4be7d58d87"
server_key_iv: "b2a687e91ed2e475492a95e004218479"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.356449 22844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 05:18:21.359746 22859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:21.361025 22844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:18:21.361188 22844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "4e02fbd7f5694a34a54f6490d90956fe"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "16cb66a353083843aeefba4be7d58d87"
server_key_iv: "b2a687e91ed2e475492a95e004218479"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.361392 22844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:21.377948 22844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:21.381651 22844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:21.381901 22844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:21.399765 22844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46647
I20260927 05:18:21.399809 22910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46647 every 8 connection(s)
I20260927 05:18:21.401032 22844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:18:21.404488 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:21.408834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22844
I20260927 05:18:21.409000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-1/wal/instance
I20260927 05:18:21.409412 388 external_mini_cluster.cc:1428] Setting key 3ce14c897922126984c59061cdffa7ad
I20260927 05:18:21.410852 22911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:21.432586 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.413403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.0.97.62:33219)
0927 05:18:21.413622 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.413629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.414657 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 05:18:21.416575 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.416581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.416646 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.416737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.418700 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.419432 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.421768 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.422169 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.422258 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.426560 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.426565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.426568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.426601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.429265 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.430198 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.430202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.430204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.430371 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.430917 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.430920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.430922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.431232 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.431351 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.432205 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.432356 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 05:18:21.434091 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.413733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.0.1:34818)
0927 05:18:21.414488 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.414531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.416293 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.416999 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.417010 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.417778 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.418520 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.418552 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.419592 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.419600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.421633 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.421650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.422384 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.422393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.422742 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.423458 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.423490 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.426332 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.429461 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.429473 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.429491 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.430087 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.430536 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.430543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.430549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.430806 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.431358 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.431369 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.431853 (+ 484us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.432149 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.432671 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":322,"threads_started":1}
I20260927 05:18:21.435840 22911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:21.451321 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.436945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.0.97.61:46647)
0927 05:18:21.437599 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.437610 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.437640 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:18:21.437692 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.437702 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.437931 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.438080 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.438716 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.440371 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.441419 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.442301 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.442456 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.445599 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.445608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.445619 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.445651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.448274 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.449081 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.449087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.449089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.449194 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.449667 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.449671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.449676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.450011 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.450162 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.450626 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.450915 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":466,"thread_start_us":311,"threads_started":1}
I20260927 05:18:21.451341 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.436537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45224)
0927 05:18:21.436783 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.436805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.436916 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.438088 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.438091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.438420 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.438543 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.438551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.440514 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.440517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.441286 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.441295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.442457 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.442461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.442659 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.443237 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.443253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.445365 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.448399 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.448407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.448413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.448961 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.449295 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.449302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.449309 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.449496 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.450142 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.450150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.450306 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.450459 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.450786 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":43}
I20260927 05:18:21.453764 22911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:21.454913 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.454361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42492)
0927 05:18:21.454574 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.454713 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:18:21.455209 22860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.456938 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.456477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42504)
0927 05:18:21.456667 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.456773 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:18:21.457187 22911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:18:21.470116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45915
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:18:21.506547 22911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 1
I20260927 05:18:21.507783 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.507264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42512)
0927 05:18:21.507474 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.507596 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:18:21.508800 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.508419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42520)
0927 05:18:21.508589 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.508676 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:18:21.509014 22911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.577251 22911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 2
I20260927 05:18:21.578526 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.577984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42522)
0927 05:18:21.578215 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.578340 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:18:21.579536 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.579172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42536)
0927 05:18:21.579314 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.579394 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 05:18:21.579747 22911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.623558 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } attempt: 5
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:21.637166 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.624522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.0.97.61:46647)
0927 05:18:21.624776 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.624782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.624805 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:18:21.624910 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.624915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.624984 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.625093 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.625909 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.626621 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.627914 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.628315 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.628466 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.631917 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.631922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.631925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.631955 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.634644 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.635335 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.635342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.635345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.635466 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.635827 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.635832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.635837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.636102 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.636253 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.636635 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.636832 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 05:18:21.637163 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.624397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45230)
0927 05:18:21.624622 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.624645 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.624778 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.625134 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.625139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.625565 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.625751 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.625761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.626767 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.626773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.627772 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.627785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.628472 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.628478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.628666 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.629430 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.629459 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.631730 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.634776 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.634782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.634786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.635222 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.635574 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.635580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.635584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.635716 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.636218 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.636223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.636382 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.636524 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.636796 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 05:18:21.638429 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:21.639477 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.639119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42544)
0927 05:18:21.639269 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.639347 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 05:18:21.640398 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.640099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42552)
0927 05:18:21.640230 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.640308 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:18:21.640594 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.670967 22911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 3
I20260927 05:18:21.672138 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.671561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42558)
0927 05:18:21.671811 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.671933 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:18:21.673235 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.672791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42570)
0927 05:18:21.673006 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.673086 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260927 05:18:21.673483 22911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.691790 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 1
I20260927 05:18:21.692792 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.692369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42584)
0927 05:18:21.692550 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.692642 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:18:21.693693 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.693402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42596)
0927 05:18:21.693536 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.693603 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:18:21.693910 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.762125 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 2
I20260927 05:18:21.763304 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.762769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42602)
0927 05:18:21.763044 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.763143 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:18:21.764302 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.763984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42610)
0927 05:18:21.764136 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.764201 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:18:21.764571 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
W20260927 05:18:21.793882 22918 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:21.794210 22918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:21.794296 22918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:21.794415 22918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:21.801304 22918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:21.801380 22918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:21.801486 22918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:21.801560 22918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:21.801661 22918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:21.810896 22918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33219,127.0.97.61:46647,127.0.97.60:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:21.812088 22918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:21.813344 22918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 05:18:21.816723 22911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 4
I20260927 05:18:21.817878 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.817335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42612)
0927 05:18:21.817583 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.817685 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:18:21.818894 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.818520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42624)
0927 05:18:21.818669 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.818759 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:18:21.819197 22911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
W20260927 05:18:21.824970 22923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:21.825059 22924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:21.825004 22926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:21.825934 22918 server_base.cc:1034] running on GCE node
I20260927 05:18:21.826622 22918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:21.827904 22918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:21.829137 22918 hybrid_clock.cc:648] HybridClock initialized: now 1790486301829075 us; error 29 us; skew 500 ppm
I20260927 05:18:21.829483 22918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:21.832896 22918 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:18:21.834208 22918 webserver.cc:460] Webserver started at http://127.0.97.60:36527/ using document root <none> and password file <none>
I20260927 05:18:21.834865 22918 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:21.834992 22918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:21.835347 22918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:21.838100 22918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data/instance:
uuid: "bb5dd59be6064fe591258458dc4f69ae"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "cea758d4264b5c3f72f818510d0c83df"
server_key_iv: "d71c8f8d73bd12240e7d51227cdf8c45"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.838835 22918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal/instance:
uuid: "bb5dd59be6064fe591258458dc4f69ae"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "cea758d4264b5c3f72f818510d0c83df"
server_key_iv: "d71c8f8d73bd12240e7d51227cdf8c45"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.843828 22918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 05:18:21.847126 22933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:21.848395 22918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:18:21.848619 22918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "bb5dd59be6064fe591258458dc4f69ae"
format_stamp: "Formatted at 2026-09-27 05:18:21 on dist-test-slave-1t6p"
server_key: "cea758d4264b5c3f72f818510d0c83df"
server_key_iv: "d71c8f8d73bd12240e7d51227cdf8c45"
server_key_version: "encryptionkey@0"
I20260927 05:18:21.848825 22918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:21.869772 22918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:21.870788 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 3
I20260927 05:18:21.871842 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.871384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42636)
0927 05:18:21.871576 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.871653 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 05:18:21.872807 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.872500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42648)
0927 05:18:21.872641 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.872711 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:18:21.873016 22837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45915: connect: Connection refused (error 111)
I20260927 05:18:21.874402 22918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:21.874666 22918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:21.892549 22918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45915
I20260927 05:18:21.892632 22984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45915 every 8 connection(s)
I20260927 05:18:21.893838 22918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:18:21.897538 22985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:21.901634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 22918
I20260927 05:18:21.901813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485951045978-388-0/minicluster-data/master-2/wal/instance
I20260927 05:18:21.902350 388 external_mini_cluster.cc:1428] Setting key e48d72fe0c61761558d2327b2726a9f5
I20260927 05:18:21.905158 22985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:21.925625 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.904371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.97.60:45915)
0927 05:18:21.905137 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.905158 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.905207 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:18:21.907285 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.907314 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.907773 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.908195 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.909887 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.911535 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.913023 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.913624 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.913750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.918082 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.918093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.918110 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.918146 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.921727 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.922516 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.922523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.922527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.922714 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.923160 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.923164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.923168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.923767 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.924076 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.924381 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.924646 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260927 05:18:21.926242 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.909391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.0.97.62:33219)
0927 05:18:21.909599 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.909605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.910021 (+ 416us) server_negotiation.cc:407] Connection header received
0927 05:18:21.910212 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.910220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.910298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.910401 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.911646 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.912323 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.913966 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.914391 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.914553 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.918518 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.918523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.918526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.918556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.921458 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.922297 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.922302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.922305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.922487 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.922917 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.922922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.922926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.923328 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.923553 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.925660 (+ 2107us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.925892 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 05:18:21.925650 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.909186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.0.1:34828)
0927 05:18:21.909864 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.909916 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.910068 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.910456 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.910467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.911163 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.911490 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.911503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.912478 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.912483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.913818 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.913832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.914622 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.914631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.914792 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.915391 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.915415 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.918325 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.921596 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.921609 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.921620 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.922169 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.922618 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.922625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.922628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.922803 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.923501 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.923510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.923806 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.924090 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.924538 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":29,"spinlock_wait_cycles":15616,"thread_start_us":424,"threads_started":1}
I20260927 05:18:21.927970 22985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:21.941318 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.928611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46647 (local address 127.0.0.1:45234)
0927 05:18:21.928884 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.928903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.929018 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.929340 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.929344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.929671 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.929796 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.929804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.930990 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.930995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.932087 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.932098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.933013 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.933018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.933196 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.933747 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.933769 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.935895 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.939066 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.939074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.939080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.939612 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.939955 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.939963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.939966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.940128 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.940587 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.940595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.940740 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.940838 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.941018 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 05:18:21.941551 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.928812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.97.61:46647)
0927 05:18:21.929001 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.929009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.929028 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:18:21.929127 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.929131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.929187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.929299 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.929935 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.930847 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.932219 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.932869 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.932997 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.936087 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.936094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.936102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.936148 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.938906 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.939724 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.939727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.939730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.939844 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.940239 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.940241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.940243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.940469 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.940624 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.940996 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.941220 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:18:21.942461 22985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:21.943423 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.928726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.0.97.62:33219)
0927 05:18:21.928884 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.928889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.928909 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:18:21.929069 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.929077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.929153 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.929219 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.930517 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.931479 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.933010 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.933615 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.933743 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.937603 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.937608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.937611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.937648 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.940494 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.941207 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.941213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.941216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.941364 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.941893 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.941897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.941899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.942289 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.942537 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.942850 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.943127 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:18:21 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:21.960269 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.943182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42666)
0927 05:18:21.943417 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:21.943437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:21.943575 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:21.943922 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:21.943925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:21.944399 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:21.944590 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.944601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.945865 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.945870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.947286 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:21.947297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.948155 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.948159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:21.948389 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.948893 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:21.948914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:21.952077 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:21.957755 (+ 5678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.957759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.957762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.958166 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.958561 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:21.958564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:21.958566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:21.958681 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:21.959275 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:21.959279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:21.959414 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 05:18:21.959580 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 05:18:21.959870 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 05:18:21.960316 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.943341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.97.60:45915)
0927 05:18:21.943489 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.943492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.943529 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:18:21.943729 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.943733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.943802 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.943898 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.944729 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.945739 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.947407 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.948023 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.948142 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.952297 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.952302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.952307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.952342 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.957621 (+ 5279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.958286 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.958291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.958295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.958443 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.958790 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.958797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.958803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.959162 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.959310 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.959653 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.960018 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":55}
I20260927 05:18:21.964555 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:21.945274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.97.61:46647)
0927 05:18:21.945450 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:21.945455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:21.945471 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:18:21.945515 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:21.945519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:21.945581 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:21.945668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:21.947161 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.948155 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.950785 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:21.951915 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:21.952241 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:21.959144 (+ 6903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:21.959149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:21.959152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:21.959182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:21.961899 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.962515 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.962521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.962527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.962694 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:21.963100 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:21.963108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:21.963111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:21.963361 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:21.963621 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:21.963931 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 05:18:21.964154 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:18:21.965760 22985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: Bootstrap starting.
I20260927 05:18:21.969370 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:21.969410 22985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:21.970458 22985 log.cc:824] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:21.973037 22985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: No bootstrap required, opened a new log
I20260927 05:18:21.971966 22812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ef88ef50994442c39f06b81142d472b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34838
I20260927 05:18:21.979877 22985 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:21.980216 22985 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5dd59be6064fe591258458dc4f69ae, State: Initialized, Role: FOLLOWER
I20260927 05:18:21.980636 22985 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:21.981515 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:21.981894 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:21.982625 22985 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:18:21.991585 23002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:21.991711 23002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:21.989466 22918 master_runner.cc:186] Error getting master registration for 127.0.97.62:33219: OK, instance_id { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" instance_seqno: 1790486300911341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:18:22.000541 22918 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:18:22.050267 22837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 4
Sep 27 05:18:22 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:18:22.066684 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.051130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42676)
0927 05:18:22.051317 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:22.051338 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:22.051459 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:22.051840 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:22.051847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:22.052462 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:22.052684 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:22.052696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.054224 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.054230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:22.055739 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:22.055755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.056755 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.056762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:22.056966 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.057624 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:22.057643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:22.059966 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:22.063946 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:22.063952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:22.063958 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:22.064421 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:22.064781 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:22.064786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:22.064789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:22.064944 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:22.065705 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:22.065709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:22.065867 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 05:18:22.066002 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 05:18:22.066282 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:18:22.066933 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.051218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.97.60:45915)
0927 05:18:22.051438 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.051444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.051468 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:18:22.051570 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.051576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.051655 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.051776 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.052845 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.054076 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.055890 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.056600 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.056723 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.060146 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:22.060153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:22.060158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:22.060199 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:22.063830 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.064529 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.064535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.064538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.064670 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.065057 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.065062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.065066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.065571 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:22.065707 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:22.066161 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 05:18:22.066371 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 05:18:22.072796 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: Bootstrap starting.
I20260927 05:18:22.075865 22837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:22.076865 22837 log.cc:824] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:22.079474 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: No bootstrap required, opened a new log
I20260927 05:18:22.084789 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:22.085180 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef88ef50994442c39f06b81142d472b2, State: Initialized, Role: FOLLOWER
I20260927 05:18:22.085688 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:22.086596 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.086968 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.087634 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:22.087765 22812 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:18:22.088016 22812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:22.091483 22812 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:22.093804 22812 leader_election.cc:288] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [CANDIDATE]: Term 1 election: Requested vote from peers 4e02fbd7f5694a34a54f6490d90956fe (127.0.97.61:46647), bb5dd59be6064fe591258458dc4f69ae (127.0.97.60:45915)
I20260927 05:18:22.094233 22960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef88ef50994442c39f06b81142d472b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb5dd59be6064fe591258458dc4f69ae"
I20260927 05:18:22.094779 22960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:22.098106 22960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef88ef50994442c39f06b81142d472b2 in term 1.
I20260927 05:18:22.098649 22786 leader_election.cc:302] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5dd59be6064fe591258458dc4f69ae, ef88ef50994442c39f06b81142d472b2; no voters:
I20260927 05:18:22.099083 23005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:22.099797 22770 master_runner.cc:186] Error getting master registration for 127.0.97.61:46647: OK, instance_id { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" instance_seqno: 1790486301394535 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:18:22.114936 22770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:18:22.118418 22911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } attempt: 5
I20260927 05:18:22.124200 23005 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [term 1 LEADER]: Becoming Leader. State: Replica: ef88ef50994442c39f06b81142d472b2, State: Running, Role: LEADER
I20260927 05:18:22.124845 23005 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
W20260927 05:18:22.126585 22939 tablet.cc:2369] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:18:22 dist-test-slave-1t6p krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 05:18:22.132321 23016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:22.132507 23016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:22.140156 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef88ef50994442c39f06b81142d472b2. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.140635 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:22.143391 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.119289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45915 (local address 127.0.0.1:42682)
0927 05:18:22.119480 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:22.119517 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:22.119691 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:22.120142 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:22.120150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:22.120832 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:22.121104 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:22.121121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.123452 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.123458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:22.125524 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:22.125542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.127938 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.127947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:22.128241 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.129140 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:22.129173 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:22.133127 (+ 3954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:22.138142 (+ 5015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:22.138150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:22.138157 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:22.138832 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:22.139469 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:22.139497 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:22.139503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:22.139764 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:22.140874 (+ 1110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:22.140902 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:22.141155 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 05:18:22.141368 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 05:18:22.142036 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 05:18:22.144358 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.119289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.97.60:45915)
0927 05:18:22.119481 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.119487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.119621 (+ 134us) server_negotiation.cc:407] Connection header received
0927 05:18:22.119864 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.119873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.119997 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.120122 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.121394 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.123248 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.125753 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.126329 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.127647 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.133326 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:22.133334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:22.133342 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:22.133385 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:22.137950 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.138975 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.138981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.139021 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.139257 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.139942 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.139970 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.139976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.140563 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:22.141850 (+ 1287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:22.142354 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 05:18:22.143161 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:18:22.147825 23025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:22.150169 23025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 05:18:22.150585 22791 tablet.cc:2369] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:18:22.151547 22911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: Bootstrap starting.
I20260927 05:18:22.155686 22911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:22.156961 22911 log.cc:824] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:22.160369 22960 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [term 1 FOLLOWER]: Refusing update from remote peer ef88ef50994442c39f06b81142d472b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:18:22.161376 23005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:18:22.164573 22911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: No bootstrap required, opened a new log
I20260927 05:18:22.191447 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.133683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.97.62:33219)
0927 05:18:22.133858 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.133864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.143259 (+ 9395us) server_negotiation.cc:407] Connection header received
0927 05:18:22.144967 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.144977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.145075 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.145156 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.160456 (+ 15300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.161820 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.166915 (+ 5095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.167631 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.167740 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.172255 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:22.172261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:22.172266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:22.172310 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:22.177834 (+ 5524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.179970 (+ 2136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.179977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.179981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.180411 (+ 430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.181383 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.181421 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.181427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.182319 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:22.184385 (+ 2066us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:22.185002 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 05:18:22.186377 (+ 1375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:18:22.197906 22911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:22.198339 22911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e02fbd7f5694a34a54f6490d90956fe, State: Initialized, Role: FOLLOWER
I20260927 05:18:22.198899 22911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } }
I20260927 05:18:22.206375 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.143858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.0.97.61:46647)
0927 05:18:22.144200 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.144208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.170389 (+ 26181us) server_negotiation.cc:407] Connection header received
0927 05:18:22.171846 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.171853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.171936 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.172005 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.175399 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.176520 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.183829 (+ 7309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.184511 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.184638 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.194241 (+ 9603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:22.194250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:22.194257 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:22.194302 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:22.198119 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.203820 (+ 5701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.203828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.203832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.204018 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.204508 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.204515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.204519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.204909 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:22.205162 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:22.205755 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 05:18:22.206011 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260927 05:18:22.211820 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.215761 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef88ef50994442c39f06b81142d472b2. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.216090 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.216306 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.214028 22911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:22.214077 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef88ef50994442c39f06b81142d472b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e02fbd7f5694a34a54f6490d90956fe"
I20260927 05:18:22.214432 22885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:22.219640 23027 mvcc.cc:205] Tried to move back new op lower bound from 7333831893634854912 to 7333831893508784128. Current Snapshot: MvccSnapshot[applied={T|T < 7333831893634854912}]
I20260927 05:18:22.220923 22885 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [term 1 FOLLOWER]: Refusing update from remote peer ef88ef50994442c39f06b81142d472b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:18:22.222463 23020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:18:22.227411 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.227689 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:22.229815 23025 catalog_manager.cc:1347] Generated new cluster ID: 84991658ed684f788acc5dc1afea976a
I20260927 05:18:22.229900 23025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:22.236115 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.236500 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.246253 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:22.170881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.0.97.60:45915)
0927 05:18:22.171129 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.171136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.186677 (+ 15541us) server_negotiation.cc:407] Connection header received
0927 05:18:22.187796 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.187806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.187916 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.188002 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.191299 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.192623 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.197073 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:22.201160 (+ 4087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:22.201295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:22.205757 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:22.205765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:22.205769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:22.205807 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:22.210199 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.211898 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.211935 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.211938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.212195 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:22.213077 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:22.213084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:22.213087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:22.213760 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:22.214499 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:22.245177 (+ 30678us) server_negotiation.cc:299] Negotiation successful
0927 05:18:22.245656 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:18:22.299383 22886 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef88ef50994442c39f06b81142d472b2 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 05:18:22.300069 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.300522 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.300848 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef88ef50994442c39f06b81142d472b2. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.301095 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:22.347441 23049 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:18:22.348924 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef88ef50994442c39f06b81142d472b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef88ef50994442c39f06b81142d472b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } peers { permanent_uuid: "4e02fbd7f5694a34a54f6490d90956fe" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46647 } } peers { permanent_uuid: "bb5dd59be6064fe591258458dc4f69ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45915 } } }
I20260927 05:18:22.349166 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2 [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:22.354554 23025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:22.359941 23025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:22.373798 23025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef88ef50994442c39f06b81142d472b2: Generated new TSK 0
I20260927 05:18:22.374876 23025 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:18:22.458353 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.455334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.0.97.62:33219)
0927 05:18:22.455592 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.455600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.456133 (+ 533us) server_negotiation.cc:407] Connection header received
0927 05:18:22.456465 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.456473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.456727 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.456829 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.458082 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50570: BlockingRecv error: recv got EOF from 127.0.0.1:50570 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 05:18:22.460781 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.456465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.97.61:46647)
0927 05:18:22.456660 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.456667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.459182 (+ 2515us) server_negotiation.cc:407] Connection header received
0927 05:18:22.459420 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.459427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.459520 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.459594 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.460568 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52890: BlockingRecv error: recv got EOF from 127.0.0.1:52890 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:18:22.464124 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.462123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.0.97.60:45915)
0927 05:18:22.462304 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.462309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.462547 (+ 238us) server_negotiation.cc:407] Connection header received
0927 05:18:22.462677 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.462683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.462786 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.462884 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.463888 (+ 1004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49268: BlockingRecv error: recv got EOF from 127.0.0.1:49268 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 05:18:22.470970 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.468837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.97.61:46647)
0927 05:18:22.468997 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.469005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.469470 (+ 465us) server_negotiation.cc:407] Connection header received
0927 05:18:22.469667 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.469672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.469780 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.469889 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.470692 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52894: BlockingRecv error: recv got EOF from 127.0.0.1:52894 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 05:18:22.480847 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.475235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.97.61:46647)
0927 05:18:22.475386 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.475391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.479519 (+ 4128us) server_negotiation.cc:407] Connection header received
0927 05:18:22.479716 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.479722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.479806 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.479909 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.480656 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52904: BlockingRecv error: recv got EOF from 127.0.0.1:52904 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 05:18:22.492903 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.488780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.0.97.61:46647)
0927 05:18:22.488950 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.488955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.491567 (+ 2612us) server_negotiation.cc:407] Connection header received
0927 05:18:22.491745 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.491749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.491837 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.491943 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.492682 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52910: BlockingRecv error: recv got EOF from 127.0.0.1:52910 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 05:18:22.502182 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.500009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.61:46647)
0927 05:18:22.500223 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.500229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.500754 (+ 525us) server_negotiation.cc:407] Connection header received
0927 05:18:22.500922 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.500929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.501041 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.501150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.501956 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52916: BlockingRecv error: recv got EOF from 127.0.0.1:52916 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 05:18:22.519537 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.517380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.97.61:46647)
0927 05:18:22.517568 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.517573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.518074 (+ 501us) server_negotiation.cc:407] Connection header received
0927 05:18:22.518286 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.518292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.518377 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.518479 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.519323 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52924: BlockingRecv error: recv got EOF from 127.0.0.1:52924 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:18:22.541150 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.538946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.0.97.61:46647)
0927 05:18:22.539165 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.539172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.539680 (+ 508us) server_negotiation.cc:407] Connection header received
0927 05:18:22.539881 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.539887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.539973 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.540091 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.540901 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52934: BlockingRecv error: recv got EOF from 127.0.0.1:52934 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 05:18:22.581928 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.579730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.0.97.61:46647)
0927 05:18:22.579897 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.579903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.580420 (+ 517us) server_negotiation.cc:407] Connection header received
0927 05:18:22.580609 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.580616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.580716 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.580840 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.581664 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52942: BlockingRecv error: recv got EOF from 127.0.0.1:52942 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 05:18:22.651086 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.649177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.0.97.61:46647)
0927 05:18:22.649365 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.649372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.649717 (+ 345us) server_negotiation.cc:407] Connection header received
0927 05:18:22.649881 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.649887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.649979 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.650107 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.650838 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52952: BlockingRecv error: recv got EOF from 127.0.0.1:52952 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 05:18:22.786271 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:22.784160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.0.97.61:46647)
0927 05:18:22.784391 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:22.784398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:22.784732 (+ 334us) server_negotiation.cc:407] Connection header received
0927 05:18:22.784952 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:22.784958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:22.785049 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:22.785159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:22.785985 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52962: BlockingRecv error: recv got EOF from 127.0.0.1:52962 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:18:22.993678 23002 catalog_manager.cc:1259] Loaded cluster ID: 84991658ed684f788acc5dc1afea976a
I20260927 05:18:22.993779 23002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: loading cluster ID for follower catalog manager: success
I20260927 05:18:22.996193 23002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: acquiring CA information for follower catalog manager: success
I20260927 05:18:22.998008 23002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb5dd59be6064fe591258458dc4f69ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:18:23.047431 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:23.045599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.0.97.61:46647)
0927 05:18:23.045825 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.045832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.046021 (+ 189us) server_negotiation.cc:407] Connection header received
0927 05:18:23.046262 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.046266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.046347 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.046458 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:23.047219 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52972: BlockingRecv error: recv got EOF from 127.0.0.1:52972 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:18:23.351895 23049 catalog_manager.cc:1259] Loaded cluster ID: 84991658ed684f788acc5dc1afea976a
I20260927 05:18:23.352046 23049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: loading cluster ID for follower catalog manager: success
I20260927 05:18:23.354657 23049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: acquiring CA information for follower catalog manager: success
I20260927 05:18:23.356429 23049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e02fbd7f5694a34a54f6490d90956fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:18:23.572077 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:23.563687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.0.97.61:46647)
0927 05:18:23.565226 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.565232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.565622 (+ 390us) server_negotiation.cc:407] Connection header received
0927 05:18:23.565855 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.565859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.566002 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.566209 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:23.566726 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.567661 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.569667 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.570101 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.570203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:23.570827 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:23.571379 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:23.571522 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:23.571652 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 05:18:23.571714 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1170,"threads_started":1}
I20260927 05:18:23.585134 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:23.578113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.0.97.60:45915)
0927 05:18:23.579301 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.579305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.579321 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:18:23.579404 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.579408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.579545 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.579679 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:23.580130 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.580802 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.582676 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.583298 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.583390 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:23.583909 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:23.584490 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:23.584617 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:23.584756 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:18:23.584817 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":958,"threads_started":1}
I20260927 05:18:23.605561 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:23.597096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.0.97.62:33219)
0927 05:18:23.597652 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.597658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.598142 (+ 484us) server_negotiation.cc:407] Connection header received
0927 05:18:23.598267 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.598273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.598490 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.598651 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:23.599395 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.600454 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.602487 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.603151 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.603277 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:23.603944 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:23.604386 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:23.604510 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:23.605158 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 05:18:23.605241 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
I20260927 05:18:23.612399 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:23.598068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.97.61:46647)
0927 05:18:23.598309 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.598317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.599625 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 05:18:23.599897 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.599905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.600015 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.600119 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:23.601441 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.602899 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.605681 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.606653 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.606779 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:23.611538 (+ 4759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:23.611660 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:23.611732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:23.611983 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 05:18:23.612084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:18:23.615679 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:23.600482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.97.60:45915)
0927 05:18:23.600645 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:23.600651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:23.605076 (+ 4425us) server_negotiation.cc:407] Connection header received
0927 05:18:23.606113 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:23.606118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:23.606218 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:23.606281 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:23.607103 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.608165 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.613107 (+ 4942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:23.613755 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:23.613858 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:23.614935 (+ 1077us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:23.615097 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:23.615197 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:23.615352 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:18:23.615431 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:18:23.615727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22770
I20260927 05:18:23.616750 22832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:23.736670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22770
I20260927 05:18:23.757997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22844
I20260927 05:18:23.759130 22906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:23.864060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22844
I20260927 05:18:23.884318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 22918
I20260927 05:18:23.885424 22980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:23.990816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 22918
2026-09-27T05:18:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4880 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:18:24 dist-test-slave-1t6p krb5kdc[23135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:18:24 dist-test-slave-1t6p krb5kdc[23135](info): set up 2 sockets
Sep 27 05:18:24 dist-test-slave-1t6p krb5kdc[23135](info): commencing operation
krb5kdc: starting...
W20260927 05:18:25.095515 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/test-user.keytab.
Sep 27 05:18:25 dist-test-slave-1t6p krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:18:25.328397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41839
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:18:25.645941 23148 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:25.646236 23148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:25.646297 23148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:25.646377 23148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:25.653292 23148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:25.653364 23148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:25.653419 23148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:25.653467 23148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:25.653538 23148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:25.662693 23148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:25.663980 23148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:25.665632 23148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:25.678591 23153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:25.679342 23154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:25.679669 23156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:25.680298 23148 server_base.cc:1034] running on GCE node
I20260927 05:18:25.681038 23148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:25.682432 23148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:25.683645 23148 hybrid_clock.cc:648] HybridClock initialized: now 1790486305683577 us; error 52 us; skew 500 ppm
I20260927 05:18:25.683974 23148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:25 dist-test-slave-1t6p krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:25.687081 23148 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:18:25.688508 23148 webserver.cc:460] Webserver started at http://127.0.97.62:44467/ using document root <none> and password file <none>
I20260927 05:18:25.689248 23148 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:25.689378 23148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:25.689695 23148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:25.692616 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data/instance:
uuid: "a02a617913f947ecb8bc0e592c34c3eb"
format_stamp: "Formatted at 2026-09-27 05:18:25 on dist-test-slave-1t6p"
server_key: "d0a514c08867f239019c67200401b8b9"
server_key_iv: "780a16525bbfc9d589e42c619b577a2c"
server_key_version: "encryptionkey@0"
I20260927 05:18:25.693372 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance:
uuid: "a02a617913f947ecb8bc0e592c34c3eb"
format_stamp: "Formatted at 2026-09-27 05:18:25 on dist-test-slave-1t6p"
server_key: "d0a514c08867f239019c67200401b8b9"
server_key_iv: "780a16525bbfc9d589e42c619b577a2c"
server_key_version: "encryptionkey@0"
I20260927 05:18:25.698199 23148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 05:18:25.701514 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:25.702732 23148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:18:25.702906 23148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal
uuid: "a02a617913f947ecb8bc0e592c34c3eb"
format_stamp: "Formatted at 2026-09-27 05:18:25 on dist-test-slave-1t6p"
server_key: "d0a514c08867f239019c67200401b8b9"
server_key_iv: "780a16525bbfc9d589e42c619b577a2c"
server_key_version: "encryptionkey@0"
I20260927 05:18:25.703182 23148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:25.728106 23148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:25.731842 23148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:25.732108 23148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:25.750396 23148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41839
I20260927 05:18:25.750417 23214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41839 every 8 connection(s)
I20260927 05:18:25.751722 23148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:18:25.755160 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:25.760460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 23148
I20260927 05:18:25.760645 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-0/wal/instance
I20260927 05:18:25.761073 388 external_mini_cluster.cc:1428] Setting key fa8f3eeaa24dd8132bb64d0a2e2b9293
I20260927 05:18:25.761453 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:25 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 05:18:25.785626 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.764483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41839 (local address 127.0.0.1:34276)
0927 05:18:25.765347 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.765412 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:25.768259 (+ 2847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:25.769180 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:25.769190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:25.769895 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:25.770219 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:25.770235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:25.772214 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:25.772222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:25.774097 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:25.774114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:25.775085 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:25.775096 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:25.775331 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:25.776018 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:25.776056 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:25.778597 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:25.781793 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:25.781803 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:25.781813 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:25.782166 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:25.782549 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:25.782553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:25.782557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:25.782700 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:25.783438 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:25.783445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:25.783643 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 05:18:25.783892 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:18:25.784421 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":125,"thread_start_us":367,"threads_started":1}
I20260927 05:18:25.785645 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.764483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.0.97.62:41839)
0927 05:18:25.766294 (+ 1811us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:25.766300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:25.766343 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:18:25.768655 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:25.768662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:25.768924 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:25.769100 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:25.770489 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:25.772056 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:25.774245 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:25.774817 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:25.774961 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:25.778813 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:25.778820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:25.778831 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:25.778873 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:25.781653 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:25.782287 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:25.782295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:25.782298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:25.782437 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:25.782798 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:25.782805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:25.782808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:25.783301 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:25.783497 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:25.784012 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 05:18:25.784285 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1632,"spinlock_wait_cycles":14080,"thread_start_us":1407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 05:18:25.788664 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:25.789757 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.789275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56556)
0927 05:18:25.789473 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.789600 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 05:18:25.790004 23166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
I20260927 05:18:25.791572 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.791124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56560)
0927 05:18:25.791300 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.791398 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 05:18:25.791756 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34859: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:18:25.814836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34859
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:18:25.841071 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } attempt: 1
I20260927 05:18:25.842134 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.841704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56564)
0927 05:18:25.841890 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.841991 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:18:25.843204 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.842790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56574)
0927 05:18:25.842933 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.843067 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:18:25.843372 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34859: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
I20260927 05:18:25.911605 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } attempt: 2
I20260927 05:18:25.912729 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.912245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56590)
0927 05:18:25.912509 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.912593 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:18:25.913717 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:25.913368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56598)
0927 05:18:25.913529 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:25.913601 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:18:25.913887 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34859: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
I20260927 05:18:26.005147 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } attempt: 3
I20260927 05:18:26.006337 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.005871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56604)
0927 05:18:26.006089 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.006196 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 05:18:26.007618 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.007240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56608)
0927 05:18:26.007402 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.007478 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:18:26.007840 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34859: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
W20260927 05:18:26.133778 23222 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:26.134140 23222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:26.134266 23222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:26.134408 23222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:26.142547 23222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:26.142668 23222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:26.142798 23222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:26.142891 23222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:26.142997 23222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:26.151105 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } attempt: 4
I20260927 05:18:26.152242 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.151724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56624)
0927 05:18:26.152024 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.152114 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 05:18:26.152851 23222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:26.153258 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.152897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56632)
0927 05:18:26.153042 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.153130 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:18:26.153424 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34859: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34859: connect: Connection refused (error 111)
I20260927 05:18:26.154213 23222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:26.155553 23222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:18:26.166953 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:26.166986 23227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:26.166919 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:26.167703 23222 server_base.cc:1034] running on GCE node
I20260927 05:18:26.168462 23222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:26.169776 23222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:26.171075 23222 hybrid_clock.cc:648] HybridClock initialized: now 1790486306170970 us; error 54 us; skew 500 ppm
I20260927 05:18:26.171389 23222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:26.174350 23222 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:18:26.175724 23222 webserver.cc:460] Webserver started at http://127.0.97.61:40823/ using document root <none> and password file <none>
I20260927 05:18:26.176370 23222 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:26.176496 23222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:26.176797 23222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:26.179608 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data/instance:
uuid: "8e1a8678d3e1452e94abaa86505362a6"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "313fab6be8878d7758b110c6175e5dd9"
server_key_iv: "761edcab497939e52f537e5747cc37a1"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.180361 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal/instance:
uuid: "8e1a8678d3e1452e94abaa86505362a6"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "313fab6be8878d7758b110c6175e5dd9"
server_key_iv: "761edcab497939e52f537e5747cc37a1"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.185014 23222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:18:26.188169 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:26.189316 23222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:18:26.189483 23222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal
uuid: "8e1a8678d3e1452e94abaa86505362a6"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "313fab6be8878d7758b110c6175e5dd9"
server_key_iv: "761edcab497939e52f537e5747cc37a1"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.189687 23222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:26.214059 23222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:26.217964 23222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:26.218237 23222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:26.236544 23222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34859
I20260927 05:18:26.236588 23288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34859 every 8 connection(s)
I20260927 05:18:26.237808 23222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:18:26.241461 23289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:26.245787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 23222
I20260927 05:18:26.245981 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-1/wal/instance
I20260927 05:18:26.246393 388 external_mini_cluster.cc:1428] Setting key 1b158141c2ada75d729b3aec3d7477f3
I20260927 05:18:26.247818 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:26.268999 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.250292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.0.97.62:41839)
0927 05:18:26.250526 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.250535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.251577 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 05:18:26.254399 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.254404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.254473 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.254583 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.256093 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.257267 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.258692 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.259441 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.259586 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.263387 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.263392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.263395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.263428 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.266139 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.266827 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.266837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.266839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.266950 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.267388 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.267393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.267396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.267666 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.267831 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.268468 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.268696 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 05:18:26.269879 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.250625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41839 (local address 127.0.0.1:34280)
0927 05:18:26.251401 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.251453 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.254109 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.254843 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.254851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.255413 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.255919 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.255934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.257423 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.257427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.258543 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.258555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.259644 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.259649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.259855 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.260525 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.260548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.263166 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.266287 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.266295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.266309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.266702 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.267080 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.267085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.267089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.267271 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.267790 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.267798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.268123 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.268363 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.268822 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
I20260927 05:18:26.271628 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:26.287894 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.272659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.97.61:34859)
0927 05:18:26.273307 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.273317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.273347 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:18:26.273401 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.273411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.273640 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.273780 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.274419 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.276114 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.277200 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.278110 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.278274 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.282139 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.282150 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.282160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.282191 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.284940 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.285737 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.285743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.285745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.285849 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.286251 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.286255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.286256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.286558 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.286710 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.287184 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.287571 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
I20260927 05:18:26.287932 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.272361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56638)
0927 05:18:26.272528 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.272543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.272662 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.273792 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.273796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.274129 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.274256 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.274263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.276273 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.276276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.277065 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.277075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.278292 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.278299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.278554 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.279166 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.279190 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.281956 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.285067 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.285074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.285081 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.285620 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.285965 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.285972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.285979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.286152 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.286683 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.286690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.286873 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.287071 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.287386 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":38}
I20260927 05:18:26.290477 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:26.291719 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.291158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53314)
0927 05:18:26.291353 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.291508 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:18:26.291978 23238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.293581 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.293144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53322)
0927 05:18:26.293318 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.293413 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:18:26.293809 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:18:26.303447 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7OtwtO/build/asan/bin/kudu
/tmp/dist-test-task7OtwtO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34711
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:18:26.343120 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 1
I20260927 05:18:26.344365 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.343879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53338)
0927 05:18:26.344136 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.344220 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 05:18:26.345327 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.344973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53340)
0927 05:18:26.345145 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.345222 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:18:26.345527 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.413733 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 2
I20260927 05:18:26.414876 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.414424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53354)
0927 05:18:26.414617 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.414705 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:18:26.415866 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.415543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53364)
0927 05:18:26.415686 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.415755 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:18:26.416076 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.452661 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } attempt: 5
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:26.467437 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.453311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56648)
0927 05:18:26.453594 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.453621 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.453774 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.454170 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.454176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.454795 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.455126 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.455147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.456180 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.456186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.457604 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.457619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.458353 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.458361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.458593 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.459286 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.459308 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.461590 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.465239 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.465243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.465245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.465604 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.466011 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.466014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.466017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.466134 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.466736 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.466738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.466857 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.466962 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.467201 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 05:18:26.467705 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.453579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.97.61:34859)
0927 05:18:26.453754 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.453759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.453778 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:18:26.453911 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.453915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.453995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.454077 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.455309 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.456027 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.457766 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.458204 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.458297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.461828 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.461835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.461840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.461878 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.465101 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.465721 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.465727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.465731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.465892 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.466249 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.466256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.466260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.466623 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.466772 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.467151 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.467403 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:18:26.468788 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:26.469892 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.469449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53380)
0927 05:18:26.469617 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.469720 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:18:26.470882 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.470554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53386)
0927 05:18:26.470718 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.470789 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:18:26.471098 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.507293 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 3
I20260927 05:18:26.508376 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.507915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53390)
0927 05:18:26.508144 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.508225 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:18:26.509359 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.508970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53394)
0927 05:18:26.509181 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.509255 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260927 05:18:26.509557 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.522308 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 1
I20260927 05:18:26.523303 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.522867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53410)
0927 05:18:26.523057 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.523140 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:18:26.524186 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.523889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53418)
0927 05:18:26.524025 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.524093 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:18:26.524377 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.592588 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 2
I20260927 05:18:26.593813 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.593334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53424)
0927 05:18:26.593566 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.593656 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:18:26.595005 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.594476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53436)
0927 05:18:26.594686 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.594803 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 05:18:26.595279 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
W20260927 05:18:26.625224 23296 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:18:26.625537 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:18:26.625602 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:18:26.625689 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:18:26.632599 23296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:18:26.632678 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:18:26.632736 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:18:26.632786 23296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:18:26.632838 23296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:18:26.642285 23296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41839,127.0.97.61:34859,127.0.97.60:34711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:18:26.643432 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:18:26.644649 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 05:18:26.652756 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 4
I20260927 05:18:26.654096 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.653531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53438)
0927 05:18:26.653773 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.653865 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:18:26.655303 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.654830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53446)
0927 05:18:26.655079 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.655167 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 05:18:26.655565 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
W20260927 05:18:26.655958 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:26.655966 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:18:26.656302 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:18:26.657608 23296 server_base.cc:1034] running on GCE node
I20260927 05:18:26.658264 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:18:26.659626 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:18:26.660873 23296 hybrid_clock.cc:648] HybridClock initialized: now 1790486306660812 us; error 37 us; skew 500 ppm
I20260927 05:18:26.661194 23296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:18:26.664239 23296 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:18:26.665653 23296 webserver.cc:460] Webserver started at http://127.0.97.60:33343/ using document root <none> and password file <none>
I20260927 05:18:26.666311 23296 fs_manager.cc:359] Metadata directory not provided
I20260927 05:18:26.666437 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:18:26.666735 23296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:18:26.669524 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data/instance:
uuid: "e6b662611895440fb0b4c280e91c28ff"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "c15151bdc7c91c7422f54fb08e314d44"
server_key_iv: "bf049ffc97ec0a8e55e1ebb2354ccf47"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.670288 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal/instance:
uuid: "e6b662611895440fb0b4c280e91c28ff"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "c15151bdc7c91c7422f54fb08e314d44"
server_key_iv: "bf049ffc97ec0a8e55e1ebb2354ccf47"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.675256 23296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:18:26.678500 23311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:18:26.679843 23296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:18:26.680023 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data,/tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal
uuid: "e6b662611895440fb0b4c280e91c28ff"
format_stamp: "Formatted at 2026-09-27 05:18:26 on dist-test-slave-1t6p"
server_key: "c15151bdc7c91c7422f54fb08e314d44"
server_key_iv: "bf049ffc97ec0a8e55e1ebb2354ccf47"
server_key_version: "encryptionkey@0"
I20260927 05:18:26.680238 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:18:26.692965 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:18:26.697115 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:18:26.697366 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:18:26.701524 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 3
I20260927 05:18:26.702620 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.702100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53460)
0927 05:18:26.702304 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.702416 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 05:18:26.703841 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.703360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53472)
0927 05:18:26.703594 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.703692 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 05:18:26.704074 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34711: connect: Connection refused (error 111)
I20260927 05:18:26.716358 23296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34711
I20260927 05:18:26.716377 23362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34711 every 8 connection(s)
I20260927 05:18:26.717696 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:18:26.721158 23363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:18:26.723546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7OtwtO/build/asan/bin/kudu as pid 23296
I20260927 05:18:26.723717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7OtwtO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485951045978-388-0/minicluster-data/master-2/wal/instance
I20260927 05:18:26.724236 388 external_mini_cluster.cc:1428] Setting key eb7b7b97ede3365e08df659aa41b676e
I20260927 05:18:26.728780 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:26.748412 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.726002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.97.60:34711)
0927 05:18:26.726768 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.726782 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.726819 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:18:26.728443 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.728464 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.728824 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.729137 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.730844 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.732396 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.733876 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.734518 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.734691 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.738839 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.738848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.738859 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.738887 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.742252 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.743801 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.743813 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.743824 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.744258 (+ 434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.744744 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.744751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.744754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.745487 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.745781 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.746368 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.746856 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":348,"threads_started":1}
I20260927 05:18:26.748708 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.731732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.0.97.62:41839)
0927 05:18:26.731954 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.731960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.732612 (+ 652us) server_negotiation.cc:407] Connection header received
0927 05:18:26.732874 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.732880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.732963 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.733068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.734069 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.734774 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.736109 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.736522 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.736659 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.740180 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.740184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.740188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.740220 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.744280 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.745552 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.745560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.745565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.745819 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.746611 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.746623 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.746629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.747235 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.747386 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.747875 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.748318 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:18:26.749022 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.731699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41839 (local address 127.0.0.1:34294)
0927 05:18:26.732485 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.732530 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.732723 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.733127 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.733131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.733672 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.733923 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.733932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.734913 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.734924 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.735954 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.735965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.736739 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.736748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.736907 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.737497 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.737530 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.739990 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.744461 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.744472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.744480 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.745106 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.745994 (+ 888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.746000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.746005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.746282 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.747369 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.747379 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.747570 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.747770 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.748243 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":26496,"thread_start_us":417,"threads_started":1}
I20260927 05:18:26.751302 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:18:26.772634 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.752061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34859 (local address 127.0.0.1:56650)
0927 05:18:26.755319 (+ 3258us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.755337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.755511 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.755927 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.755933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.756422 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.756625 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.756641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.758095 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.758102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.759440 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.759451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.760141 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.760146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.760318 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.760918 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.760935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.763084 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.769493 (+ 6409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.769503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.769526 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.770252 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.770691 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.770695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.770697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.770839 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.771661 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.771666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.771815 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.771985 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.772267 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3136}
I20260927 05:18:26.772729 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.752431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.0.97.61:34859)
0927 05:18:26.753015 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.753023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.755412 (+ 2389us) server_negotiation.cc:407] Connection header received
0927 05:18:26.755687 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.755694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.755778 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.755878 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.756778 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.757923 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.759594 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.759997 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.760133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.763312 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.763321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.763331 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.763382 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.769340 (+ 5958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.770395 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.770403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.770410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.770574 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.770977 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.770983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.770986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.771423 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.771597 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.772125 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.772353 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260927 05:18:26.774349 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:18:26.776558 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.757189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.0.97.62:41839)
0927 05:18:26.757361 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.757365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.757379 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:18:26.757417 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.757420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.757474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.757582 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.759015 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.760179 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.761895 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.762538 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.762648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.769509 (+ 6861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.769516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.769522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.769569 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.773346 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.774160 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.774164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.774166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.774351 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.774796 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.774802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.774806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.775212 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.775476 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.776035 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.776249 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:18:26.791779 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.775133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53482)
0927 05:18:26.775307 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.775326 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.775415 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.775856 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.775860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.776315 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.776561 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.776575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.778015 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.778024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.779671 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.779691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.780930 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.780949 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.781192 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.781804 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.781878 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.784929 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.789257 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.789263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.789267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.789675 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.790006 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.790014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.790017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.790139 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.790734 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.790738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.790871 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.791094 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.791386 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:18:26.791842 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.775275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.0.97.60:34711)
0927 05:18:26.775433 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.775439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.775461 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:18:26.775506 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.775511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.775589 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.775687 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.776697 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.777828 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.779832 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.780587 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.780702 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.785201 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.785235 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.785243 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.785282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.789156 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.789788 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.789794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.789799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.789920 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.790248 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.790255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.790259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.790641 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.790771 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.791193 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.791548 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":49}
I20260927 05:18:26.795300 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.778182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.0.97.61:34859)
0927 05:18:26.778337 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.778343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.778359 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:18:26.778436 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.778443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.778509 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.778614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.779814 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.780493 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.781751 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.782173 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.782324 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.787888 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.787897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.787904 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.787964 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.792224 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.792877 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.792884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.792889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.793067 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.793591 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.793598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.793601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.794022 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.794330 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.794618 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.794874 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:18:26.798316 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: Bootstrap starting.
I20260927 05:18:26.799925 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:18:26.802214 23363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:26.803457 23363 log.cc:824] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:26.802975 23190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a02a617913f947ecb8bc0e592c34c3eb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34306
I20260927 05:18:26.806082 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: No bootstrap required, opened a new log
I20260927 05:18:26.813036 23363 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:26.813347 23363 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b662611895440fb0b4c280e91c28ff, State: Initialized, Role: FOLLOWER
I20260927 05:18:26.813786 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:26.814656 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:26.815047 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:26.815742 23363 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:18:26.824832 23380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:26.825011 23380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:26.822753 23296 master_runner.cc:186] Error getting master registration for 127.0.97.62:41839: OK, instance_id { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" instance_seqno: 1790486305745011 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:18:26.830648 23296 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:18:26.881299 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 4
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:18:26.897050 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.882321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.0.97.60:34711)
0927 05:18:26.882523 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.882527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.882545 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:18:26.882598 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.882601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.882659 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.882744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.883722 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.884847 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.886623 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.887172 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.887313 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.891057 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.891063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.891071 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.891113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.894434 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.895105 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.895110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.895114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.895239 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.895616 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.895623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.895626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.895947 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.896127 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.896469 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.896680 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:18:26.897044 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.882116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53496)
0927 05:18:26.882322 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.882344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.882475 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.882789 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.882794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.883339 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.883543 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.883557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.884985 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.884991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.886480 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.886496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.887329 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.887335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.887563 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.888209 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.888232 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.890861 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.894563 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.894569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.894573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.894938 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.895351 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.895355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.895359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.895513 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.896070 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.896075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.896237 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.896370 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.896657 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:18:26.902885 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: Bootstrap starting.
I20260927 05:18:26.906984 23215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:26.908303 23215 log.cc:824] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:26.911514 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: No bootstrap required, opened a new log
I20260927 05:18:26.918947 23215 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:26.919359 23215 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02a617913f947ecb8bc0e592c34c3eb, State: Initialized, Role: FOLLOWER
I20260927 05:18:26.919831 23215 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:26.920786 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:26.921159 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:26.921924 23215 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:26.921957 23190 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:18:26.922291 23190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:26.925053 23190 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:26.927228 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a02a617913f947ecb8bc0e592c34c3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e6b662611895440fb0b4c280e91c28ff"
I20260927 05:18:26.927801 23338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:26.931137 23338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02a617913f947ecb8bc0e592c34c3eb in term 1.
I20260927 05:18:26.931739 23164 leader_election.cc:302] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02a617913f947ecb8bc0e592c34c3eb, e6b662611895440fb0b4c280e91c28ff; no voters:
I20260927 05:18:26.932155 23383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:18:26.932207 23190 leader_election.cc:288] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [CANDIDATE]: Term 1 election: Requested vote from peers 8e1a8678d3e1452e94abaa86505362a6 (127.0.97.61:34859), e6b662611895440fb0b4c280e91c28ff (127.0.97.60:34711)
I20260927 05:18:26.945185 23383 raft_consensus.cc:686] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [term 1 LEADER]: Becoming Leader. State: Replica: a02a617913f947ecb8bc0e592c34c3eb, State: Running, Role: LEADER
I20260927 05:18:26.946314 23383 consensus_queue.cc:237] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
W20260927 05:18:26.949222 23317 tablet.cc:2369] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:18:26.954766 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } attempt: 5
I20260927 05:18:26.962765 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02a617913f947ecb8bc0e592c34c3eb. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:26.963340 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: This master's current role is: LEADER
Sep 27 05:18:26 dist-test-slave-1t6p krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790486306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:18:26.973291 23398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:18:26.976637 23400 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:18:26.980391 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.955532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34711 (local address 127.0.0.1:53498)
0927 05:18:26.955972 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 05:18:26.955992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:18:26.956122 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:18:26.957014 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:18:26.957019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:18:26.957573 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:18:26.957782 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.957798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.959283 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.959289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.960735 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:18:26.960749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.961706 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.961712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:18:26.961919 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.962800 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:18:26.962828 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:18:26.967416 (+ 4588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:18:26.973965 (+ 6549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.973970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.973973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.974757 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.975521 (+ 764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:18:26.975526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:18:26.975530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:18:26.975821 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:18:26.977301 (+ 1480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:18:26.977309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:18:26.977623 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 05:18:26.977840 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 05:18:26.978477 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260927 05:18:26.981240 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.956350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.97.60:34711)
0927 05:18:26.956529 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.956537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:26.956557 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:18:26.956615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:26.956619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:26.956694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:26.956801 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:26.957972 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.959120 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.960914 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:26.961574 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:26.961687 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:26.967725 (+ 6038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:26.967734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:26.967742 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:26.967792 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:26.973765 (+ 5973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.975041 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.975047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.975050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.975325 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:26.976102 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:26.976112 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:26.976118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:26.976844 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:26.978429 (+ 1585us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:26.978956 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 05:18:26.980033 (+ 1077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 05:18:26.984671 23169 tablet.cc:2369] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:18:26.986799 23289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: Bootstrap starting.
I20260927 05:18:26.991037 23289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: Neither blocks nor log segments found. Creating new log.
I20260927 05:18:26.980711 23148 master_runner.cc:186] Error getting master registration for 127.0.97.61:34859: OK, instance_id { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" instance_seqno: 1790486306231488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:18:26.991583 23148 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:18:26.994513 23289 log.cc:824] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:18:27.005546 23398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:18:27.014034 23289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: No bootstrap required, opened a new log
I20260927 05:18:27.041570 23289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:27.042057 23289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1a8678d3e1452e94abaa86505362a6, State: Initialized, Role: FOLLOWER
I20260927 05:18:27.042649 23289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } }
I20260927 05:18:27.043130 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.993208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.62:41839)
0927 05:18:26.993362 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.993368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.001566 (+ 8198us) server_negotiation.cc:407] Connection header received
0927 05:18:27.002024 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.002031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.002114 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.002190 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.011046 (+ 8856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.012267 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.016789 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.017565 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.017681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:27.030147 (+ 12466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:27.030154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:27.030161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:27.030204 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:27.034162 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.035074 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.035081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.035087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.035260 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.035754 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.035761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.035764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.036163 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:27.036386 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:27.042412 (+ 6026us) server_negotiation.cc:299] Negotiation successful
0927 05:18:27.042670 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:18:27.046245 23338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [term 1 FOLLOWER]: Refusing update from remote peer a02a617913f947ecb8bc0e592c34c3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:18:27.047647 23390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:18:27.048349 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:26.992775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.97.61:34859)
0927 05:18:26.992957 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:26.992964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.005111 (+ 12147us) server_negotiation.cc:407] Connection header received
0927 05:18:27.005741 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.005748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.005829 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.005902 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.014270 (+ 8368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.015450 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.022417 (+ 6967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.023288 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.023409 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:27.036961 (+ 13552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:27.036969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:27.036974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:27.037018 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:27.040962 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.043427 (+ 2465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.043433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.043437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.043603 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.046546 (+ 2943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.046552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.046555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.046906 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:27.047167 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:27.047727 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 05:18:27.047974 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:18:27.062078 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:27.005237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.97.60:34711)
0927 05:18:27.005442 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.005448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.042612 (+ 37164us) server_negotiation.cc:407] Connection header received
0927 05:18:27.045100 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.045106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.045198 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.045271 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.047330 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.048413 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.050468 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:27.051181 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:27.051302 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:27.054516 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:18:27.054522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:18:27.054527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:18:27.054566 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:18:27.058847 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.059669 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.059675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.059680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.059821 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:18:27.060272 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:18:27.060278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:18:27.060281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:18:27.060636 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:18:27.060879 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:27.061435 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 05:18:27.061669 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:18:27.060675 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.065179 23289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:18:27.065219 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a02a617913f947ecb8bc0e592c34c3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e1a8678d3e1452e94abaa86505362a6"
I20260927 05:18:27.065673 23264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:18:27.069859 23264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02a617913f947ecb8bc0e592c34c3eb in term 1.
I20260927 05:18:27.070833 23263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [term 1 FOLLOWER]: Refusing update from remote peer a02a617913f947ecb8bc0e592c34c3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:18:27.063158 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:27.072099 23390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:18:27.078415 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02a617913f947ecb8bc0e592c34c3eb. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.078887 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:27.087683 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.088047 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:27.092123 23398 catalog_manager.cc:1347] Generated new cluster ID: 26cb0242b6044a2b9e1e2138bf4ce495
I20260927 05:18:27.092221 23398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:18:27.097592 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.097821 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:27.104496 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02a617913f947ecb8bc0e592c34c3eb. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.104826 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:18:27.106400 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.106657 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb [sys.catalog]: This master's current role is: LEADER
I20260927 05:18:27.108338 23425 mvcc.cc:205] Tried to move back new op lower bound from 7333831913582137344 to 7333831913263738880. Current Snapshot: MvccSnapshot[applied={T|T < 7333831913582137344}]
I20260927 05:18:27.112864 23398 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:18:27.114705 23398 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:18:27.121279 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a02a617913f947ecb8bc0e592c34c3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a02a617913f947ecb8bc0e592c34c3eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41839 } } peers { permanent_uuid: "8e1a8678d3e1452e94abaa86505362a6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34859 } } peers { permanent_uuid: "e6b662611895440fb0b4c280e91c28ff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34711 } } }
I20260927 05:18:27.121599 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 05:18:27.134024 23421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:18:27.134155 23421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:18:27.139920 23398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a02a617913f947ecb8bc0e592c34c3eb: Generated new TSK 0
I20260927 05:18:27.140928 23398 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:18:27.254014 23218 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.237810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.97.62:41839)
0927 05:18:27.238041 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.238050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.251500 (+ 13450us) server_negotiation.cc:407] Connection header received
0927 05:18:27.252012 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.252021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.252138 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.252237 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.253677 (+ 1440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34328: BlockingRecv error: recv got EOF from 127.0.0.1:34328 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 05:18:27.256528 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.243946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.97.61:34859)
0927 05:18:27.244127 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.244134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.254744 (+ 10610us) server_negotiation.cc:407] Connection header received
0927 05:18:27.255044 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.255052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.255149 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.255232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.256269 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56682: BlockingRecv error: recv got EOF from 127.0.0.1:56682 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 05:18:27.258978 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.251665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.97.60:34711)
0927 05:18:27.251852 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.251857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.257156 (+ 5299us) server_negotiation.cc:407] Connection header received
0927 05:18:27.257409 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.257414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.257511 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.257599 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.258684 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53524: BlockingRecv error: recv got EOF from 127.0.0.1:53524 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 05:18:27.297844 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.287909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.0.97.61:34859)
0927 05:18:27.288098 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.288105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.296009 (+ 7904us) server_negotiation.cc:407] Connection header received
0927 05:18:27.296325 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.296331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.296480 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.296559 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.297566 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56698: BlockingRecv error: recv got EOF from 127.0.0.1:56698 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:18:27.336265 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.327048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.97.61:34859)
0927 05:18:27.327226 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.327232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.334424 (+ 7192us) server_negotiation.cc:407] Connection header received
0927 05:18:27.334869 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.334875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.334966 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.335073 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.336040 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56706: BlockingRecv error: recv got EOF from 127.0.0.1:56706 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 05:18:27.381773 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.372547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.0.97.61:34859)
0927 05:18:27.372734 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.372740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.379924 (+ 7184us) server_negotiation.cc:407] Connection header received
0927 05:18:27.380366 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.380372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.380492 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.380568 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.381517 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 05:18:27.429692 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.420286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.97.61:34859)
0927 05:18:27.420528 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.420534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.427862 (+ 7328us) server_negotiation.cc:407] Connection header received
0927 05:18:27.428309 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.428317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.428431 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.428508 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.429456 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56720: BlockingRecv error: recv got EOF from 127.0.0.1:56720 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260927 05:18:27.478982 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.476646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.97.61:34859)
0927 05:18:27.476843 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.476853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.477422 (+ 569us) server_negotiation.cc:407] Connection header received
0927 05:18:27.477626 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.477633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.477735 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.477875 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.478728 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56736: BlockingRecv error: recv got EOF from 127.0.0.1:56736 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 05:18:27.500463 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.498104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.97.61:34859)
0927 05:18:27.498286 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.498292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.498820 (+ 528us) server_negotiation.cc:407] Connection header received
0927 05:18:27.499058 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.499065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.499172 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.499306 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.500161 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56742: BlockingRecv error: recv got EOF from 127.0.0.1:56742 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 05:18:27.538517 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.536138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.97.61:34859)
0927 05:18:27.536325 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.536331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.536896 (+ 565us) server_negotiation.cc:407] Connection header received
0927 05:18:27.537099 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.537106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.537226 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.537360 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.538244 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56750: BlockingRecv error: recv got EOF from 127.0.0.1:56750 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 05:18:27.608485 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.606305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.97.61:34859)
0927 05:18:27.606579 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.606589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.607039 (+ 450us) server_negotiation.cc:407] Connection header received
0927 05:18:27.607236 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.607240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.607311 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.607403 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.608269 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56758: BlockingRecv error: recv got EOF from 127.0.0.1:56758 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 05:18:27.742659 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:27.740636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.0.97.61:34859)
0927 05:18:27.740918 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:27.740926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:27.741227 (+ 301us) server_negotiation.cc:407] Connection header received
0927 05:18:27.741444 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:27.741456 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:27.741535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:27.741642 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:27.742367 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56768: BlockingRecv error: recv got EOF from 127.0.0.1:56768 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 05:18:27.826874 23380 catalog_manager.cc:1259] Loaded cluster ID: 26cb0242b6044a2b9e1e2138bf4ce495
I20260927 05:18:27.826974 23380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: loading cluster ID for follower catalog manager: success
I20260927 05:18:27.829509 23380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: acquiring CA information for follower catalog manager: success
I20260927 05:18:27.831554 23380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6b662611895440fb0b4c280e91c28ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:18:28.005637 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:18:28.003940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.97.61:34859)
0927 05:18:28.004144 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.004151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.004265 (+ 114us) server_negotiation.cc:407] Connection header received
0927 05:18:28.004518 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.004522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.004597 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.004711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:18:28.005463 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56774: BlockingRecv error: recv got EOF from 127.0.0.1:56774 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:18:28.135934 23421 catalog_manager.cc:1259] Loaded cluster ID: 26cb0242b6044a2b9e1e2138bf4ce495
I20260927 05:18:28.136039 23421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: loading cluster ID for follower catalog manager: success
I20260927 05:18:28.138446 23421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: acquiring CA information for follower catalog manager: success
I20260927 05:18:28.140213 23421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e1a8678d3e1452e94abaa86505362a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:18:28.529088 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:28.523260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.97.61:34859)
0927 05:18:28.523783 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.523790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.523927 (+ 137us) server_negotiation.cc:407] Connection header received
0927 05:18:28.524149 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.524155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.524314 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.524455 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:28.524872 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.525599 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.526924 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.527338 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.527481 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:28.527967 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:28.528507 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:28.528667 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:28.528724 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 05:18:28.528803 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
I20260927 05:18:28.540032 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:28.534759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.0.97.60:34711)
0927 05:18:28.535227 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.535232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.535248 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:18:28.535394 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.535398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.535561 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.535691 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:28.536013 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.536709 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.537974 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.538375 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.538474 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:28.538893 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:28.539396 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:28.539516 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:28.539626 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 05:18:28.539684 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":281,"threads_started":1}
I20260927 05:18:28.553381 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:28.545847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.97.61:34859)
0927 05:18:28.546046 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.546055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.547411 (+ 1356us) server_negotiation.cc:407] Connection header received
0927 05:18:28.547499 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.547519 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.547609 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.547707 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:28.548395 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.549408 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.551481 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.552018 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.552154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:28.552668 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:28.552778 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:28.552900 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:28.553093 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 05:18:28.553182 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:18:28.553954 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:28.547023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.97.60:34711)
0927 05:18:28.547217 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.547223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.548221 (+ 998us) server_negotiation.cc:407] Connection header received
0927 05:18:28.548404 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.548409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.548491 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.548550 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:28.549251 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.550113 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.551827 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.552396 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.552515 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:28.553231 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:28.553340 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:28.553399 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:28.553623 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 05:18:28.553692 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:18:28.554690 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:18:28.545768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.97.62:41839)
0927 05:18:28.547817 (+ 2049us) server_negotiation.cc:206] Beginning negotiation
0927 05:18:28.547824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:18:28.547844 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:18:28.547966 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:18:28.547971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:18:28.548128 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:18:28.548318 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:18:28.548800 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.549691 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.551710 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:18:28.552345 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:18:28.552988 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:18:28.553262 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:18:28.553670 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:18:28.553768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:18:28.554345 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 05:18:28.554441 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1901,"thread_start_us":1772,"threads_started":1}
I20260927 05:18:28.559794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 23148
I20260927 05:18:28.560683 23210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:28.669246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 23148
W20260927 05:18:28.691589 23239 connection.cc:535] server connection from 127.0.0.1:56648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 05:18:28.691725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 23222
I20260927 05:18:28.692808 23284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:28.796415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 23222
I20260927 05:18:28.816205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7OtwtO/build/asan/bin/kudu pid 23296
I20260927 05:18:28.817333 23358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:18:28.921929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7OtwtO/build/asan/bin/kudu with pid 23296
2026-09-27T05:18:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4930 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9810 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357841 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST